[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:27:26 dist-test-slave-7vjv krb5kdc[9633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:27:26 dist-test-slave-7vjv krb5kdc[9633](info): set up 2 sockets
Aug 09 09:27:26 dist-test-slave-7vjv krb5kdc[9633](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 09:27:27.883430 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:27:27 dist-test-slave-7vjv krb5kdc[9633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:27:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:27:27.995247 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:35859
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:35859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:28.094681 9653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:28.094867 9653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:28.094899 9653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:28.098119 9653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:28.098160 9653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:28.098176 9653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:28.098191 9653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:28.098203 9653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:28.102747 9653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:35859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:35859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:28.103436 9653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:28.104225 9653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:28.111900 9665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:28.111882 9663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:28.111883 9662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:28.112010 9653 server_base.cc:1034] running on GCE node
I20260809 09:27:28.112692 9653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:28.113655 9653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:28.114851 9653 hybrid_clock.cc:648] HybridClock initialized: now 1786267648114849 us; error 52 us; skew 500 ppm
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[9633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:28.117465 9653 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:27:28.118197 9653 webserver.cc:460] Webserver started at http://127.9.102.62:43015/ using document root <none> and password file <none>
I20260809 09:27:28.118623 9653 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:28.118674 9653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:28.118809 9653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:28.120743 9653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "6690df7a84de46f79824541df416be16"
format_stamp: "Formatted at 2026-08-09 09:27:28 on dist-test-slave-7vjv"
server_key: "a89235cd821e7227804882cc1f266f00"
server_key_iv: "9e3e79e6ae35c96833d6bb13b08c3df1"
server_key_version: "encryptionkey@0"
I20260809 09:27:28.121239 9653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "6690df7a84de46f79824541df416be16"
format_stamp: "Formatted at 2026-08-09 09:27:28 on dist-test-slave-7vjv"
server_key: "a89235cd821e7227804882cc1f266f00"
server_key_iv: "9e3e79e6ae35c96833d6bb13b08c3df1"
server_key_version: "encryptionkey@0"
I20260809 09:27:28.124351 9653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 09:27:28.126329 9682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:28.127288 9653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:28.127413 9653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "6690df7a84de46f79824541df416be16"
format_stamp: "Formatted at 2026-08-09 09:27:28 on dist-test-slave-7vjv"
server_key: "a89235cd821e7227804882cc1f266f00"
server_key_iv: "9e3e79e6ae35c96833d6bb13b08c3df1"
server_key_version: "encryptionkey@0"
I20260809 09:27:28.127503 9653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:28.135809 9653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:28.138203 9653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:28.138358 9653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:28.144678 9653 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:35859
I20260809 09:27:28.144690 9777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:35859 every 8 connection(s)
I20260809 09:27:28.145495 9653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:28.147799 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:28.150249 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 9653
I20260809 09:27:28.150391 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:28.151427 9624 external_mini_cluster.cc:1428] Setting key 82b81fe7a834580daa62a8e6350c452a
I20260809 09:27:28.152758 9778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16: Bootstrap starting.
I20260809 09:27:28.155251 9778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:28.155920 9778 log.cc:824] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:28.157650 9778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16: No bootstrap required, opened a new log
I20260809 09:27:28.160935 9778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6690df7a84de46f79824541df416be16" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 35859 } }
I20260809 09:27:28.161175 9778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:28.161252 9778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6690df7a84de46f79824541df416be16, State: Initialized, Role: FOLLOWER
I20260809 09:27:28.161650 9778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6690df7a84de46f79824541df416be16" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 35859 } }
I20260809 09:27:28.161755 9778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:28.161835 9778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:28.161926 9778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:28.162643 9778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6690df7a84de46f79824541df416be16" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 35859 } }
I20260809 09:27:28.162935 9778 leader_election.cc:302] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6690df7a84de46f79824541df416be16; no voters:
I20260809 09:27:28.163221 9778 leader_election.cc:288] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:28.163304 9786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:28.163468 9786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [term 1 LEADER]: Becoming Leader. State: Replica: 6690df7a84de46f79824541df416be16, State: Running, Role: LEADER
I20260809 09:27:28.163731 9786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6690df7a84de46f79824541df416be16" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 35859 } }
I20260809 09:27:28.164292 9778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:28.165288 9787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6690df7a84de46f79824541df416be16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6690df7a84de46f79824541df416be16" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 35859 } } }
I20260809 09:27:28.165433 9787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:28.165668 9788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6690df7a84de46f79824541df416be16. Latest consensus state: current_term: 1 leader_uuid: "6690df7a84de46f79824541df416be16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6690df7a84de46f79824541df416be16" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 35859 } } }
I20260809 09:27:28.165762 9788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16 [sys.catalog]: This master's current role is: LEADER
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
W20260809 09:27:28.169250 9805 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:27:28.169324 9805 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:27:28.169409 9798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:28.170199 9798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:28.173341 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:28.154028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.9.102.62:35859)
0809 09:27:28.154373 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:28.154388 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:28.155396 (+ 1008us) server_negotiation.cc:407] Connection header received
0809 09:27:28.158336 (+ 2940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:28.158350 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:28.158602 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:28.158912 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:28.160946 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.162183 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.163288 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.163854 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.163984 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:28.168362 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:28.168366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:28.168376 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:28.168398 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:28.170092 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:28.170652 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:28.170657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:28.170660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:28.170779 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:28.171120 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:28.171123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:28.171124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:28.171455 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:28.171651 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:28.172278 (+ 627us) server_negotiation.cc:299] Negotiation successful
0809 09:27:28.172546 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":100,"threads_started":1}
I20260809 09:27:28.175756 9798 catalog_manager.cc:1347] Generated new cluster ID: 2361587cfe9146f7b60298b2e6793250
I20260809 09:27:28.175810 9798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:28.187194 9798 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:28.187947 9798 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:28.201068 9798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6690df7a84de46f79824541df416be16: Generated new TSK 0
I20260809 09:27:28.201588 9798 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:27:28.232594 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:35859
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:28.348029 9812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:28.348337 9812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:28.348373 9812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:28.351910 9812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:28.351949 9812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:28.351994 9812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:27:28.356508 9812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:35859
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:28.357275 9812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:28.358062 9812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:28.366717 9827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:28.366752 9826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:28.366758 9830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:28.366752 9812 server_base.cc:1034] running on GCE node
I20260809 09:27:28.367388 9812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:28.367883 9812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:28.369050 9812 hybrid_clock.cc:648] HybridClock initialized: now 1786267648369040 us; error 33 us; skew 500 ppm
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[9633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:28.371431 9812 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:27:28.372222 9812 webserver.cc:460] Webserver started at http://127.9.102.1:43801/ using document root <none> and password file <none>
I20260809 09:27:28.372768 9812 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:28.372855 9812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:28.373064 9812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:28.374779 9812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "9e36654e00d545039b95973d5dcf24f7"
format_stamp: "Formatted at 2026-08-09 09:27:28 on dist-test-slave-7vjv"
server_key: "056e492f82a68fea97202e8fa87d35c1"
server_key_iv: "4bd857cc674ebcfd4002a3e25d8cfb8c"
server_key_version: "encryptionkey@0"
I20260809 09:27:28.375267 9812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "9e36654e00d545039b95973d5dcf24f7"
format_stamp: "Formatted at 2026-08-09 09:27:28 on dist-test-slave-7vjv"
server_key: "056e492f82a68fea97202e8fa87d35c1"
server_key_iv: "4bd857cc674ebcfd4002a3e25d8cfb8c"
server_key_version: "encryptionkey@0"
I20260809 09:27:28.378437 9812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260809 09:27:28.380566 9838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:28.381672 9812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:28.381786 9812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "9e36654e00d545039b95973d5dcf24f7"
format_stamp: "Formatted at 2026-08-09 09:27:28 on dist-test-slave-7vjv"
server_key: "056e492f82a68fea97202e8fa87d35c1"
server_key_iv: "4bd857cc674ebcfd4002a3e25d8cfb8c"
server_key_version: "encryptionkey@0"
I20260809 09:27:28.381876 9812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:28.395787 9812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:28.398079 9812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:28.398274 9812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:28.398783 9812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:28.399493 9812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:28.399561 9812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:28.399636 9812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:28.399677 9812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:28.407894 9812 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:40149
I20260809 09:27:28.407938 10034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:40149 every 8 connection(s)
I20260809 09:27:28.408676 9812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:27:28.415534 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 9812
I20260809 09:27:28.415676 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:27:28.415910 9624 external_mini_cluster.cc:1428] Setting key 2f446305a88ca5c0bd0a04a582571feb
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:28.422058 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:28.410415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:46847 (local address 127.9.102.62:35859)
0809 09:27:28.410540 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:28.410544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:28.411245 (+ 701us) server_negotiation.cc:407] Connection header received
0809 09:27:28.412519 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:28.412523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:28.412581 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:28.412688 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:28.413803 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.414347 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.415573 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.415882 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.415989 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:28.418427 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:28.418434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:28.418439 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:28.418473 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:28.419966 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:28.420605 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:28.420607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:28.420608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:28.420683 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:28.421071 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:28.421073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:28.421074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:28.421222 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:28.421290 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:28.421844 (+ 554us) server_negotiation.cc:299] Negotiation successful
0809 09:27:28.421946 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 09:27:28.423162 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:28.410726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:35859 (local address 127.9.102.1:46847)
0809 09:27:28.411123 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:28.411160 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:28.412325 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:28.412842 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:28.412848 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:28.413227 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:28.413655 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:28.413669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.414475 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.414483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:28.415399 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:28.415414 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.416088 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.416114 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:28.416292 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:28.416908 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:28.416923 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:28.418274 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:28.420087 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:28.420122 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:28.420138 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:28.420507 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:28.420775 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:28.420781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:28.420785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:28.420964 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:28.421316 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:28.421324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:28.421600 (+ 276us) client_negotiation.cc:769] Sending connection context
0809 09:27:28.421784 (+ 184us) client_negotiation.cc:240] Negotiation successful
0809 09:27:28.422153 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":127,"threads_started":1}
I20260809 09:27:28.424439 10035 heartbeater.cc:347] Connected to a master server at 127.9.102.62:35859
I20260809 09:27:28.424676 10035 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:28.425099 10035 heartbeater.cc:511] Master 127.9.102.62:35859 requested a full tablet report, sending...
I20260809 09:27:28.426651 9719 ts_manager.cc:194] Registered new tserver with Master: 9e36654e00d545039b95973d5dcf24f7 (127.9.102.1:40149)
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:27:28.427846 9719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:46847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:27:28.442687 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:35859
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:28.543138 10044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:28.543332 10044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:28.543368 10044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:28.546903 10044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:28.546948 10044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:28.547037 10044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:27:28.551940 10044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:35859
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:28.552861 10044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:28.553655 10044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 09:27:28.561743 10044 server_base.cc:1034] running on GCE node
W20260809 09:27:28.561700 10053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:28.561704 10056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:28.561715 10051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:28.562284 10044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:28.562815 10044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:28.563998 10044 hybrid_clock.cc:648] HybridClock initialized: now 1786267648563973 us; error 40 us; skew 500 ppm
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[9633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:28.566670 10044 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:27:28.567395 10044 webserver.cc:460] Webserver started at http://127.9.102.2:41079/ using document root <none> and password file <none>
I20260809 09:27:28.567852 10044 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:28.567931 10044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:28.568185 10044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:28.569900 10044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "fb065bca04bd47a7a41cdfc3c1a389f2"
format_stamp: "Formatted at 2026-08-09 09:27:28 on dist-test-slave-7vjv"
server_key: "540c8b85366b8b6b52cf0d9500a2ddd0"
server_key_iv: "7416388d5ab4c59dc854c438d2a66a4a"
server_key_version: "encryptionkey@0"
I20260809 09:27:28.570397 10044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "fb065bca04bd47a7a41cdfc3c1a389f2"
format_stamp: "Formatted at 2026-08-09 09:27:28 on dist-test-slave-7vjv"
server_key: "540c8b85366b8b6b52cf0d9500a2ddd0"
server_key_iv: "7416388d5ab4c59dc854c438d2a66a4a"
server_key_version: "encryptionkey@0"
I20260809 09:27:28.573522 10044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:28.575417 10066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:28.576467 10044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:28.576609 10044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "fb065bca04bd47a7a41cdfc3c1a389f2"
format_stamp: "Formatted at 2026-08-09 09:27:28 on dist-test-slave-7vjv"
server_key: "540c8b85366b8b6b52cf0d9500a2ddd0"
server_key_iv: "7416388d5ab4c59dc854c438d2a66a4a"
server_key_version: "encryptionkey@0"
I20260809 09:27:28.576719 10044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:28.594813 10044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:28.597687 10044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:28.597852 10044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:28.600175 10044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:28.600853 10044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:28.600915 10044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:28.600981 10044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:28.601019 10044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:28.609018 10044 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:41381
I20260809 09:27:28.609040 10250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:41381 every 8 connection(s)
I20260809 09:27:28.609761 10044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:28.618243 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 10044
I20260809 09:27:28.618360 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:27:28.618618 9624 external_mini_cluster.cc:1428] Setting key 7e26a1af1c41a14178e527bf2a88f7fa
I20260809 09:27:28.621846 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:28.611368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:41469 (local address 127.9.102.62:35859)
0809 09:27:28.611514 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:28.611518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:28.612272 (+ 754us) server_negotiation.cc:407] Connection header received
0809 09:27:28.613358 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:28.613360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:28.613403 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:28.613471 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:28.614393 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.614856 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.615811 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.615992 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.616062 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:28.618341 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:28.618346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:28.618349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:28.618375 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:28.619806 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:28.620365 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:28.620368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:28.620371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:28.620461 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:28.620766 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:28.620771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:28.620776 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:28.620999 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:28.621092 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:28.621482 (+ 390us) server_negotiation.cc:299] Negotiation successful
0809 09:27:28.621619 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 09:27:28.622284 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:28.611689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:35859 (local address 127.9.102.2:41469)
0809 09:27:28.612143 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:28.612190 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:28.613195 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:28.613615 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:28.613620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:28.613898 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:28.614290 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:28.614300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.614947 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.614950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:28.615699 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:28.615707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.616131 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.616138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:28.616317 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:28.616841 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:28.616856 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:28.618173 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:28.619930 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:28.619936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:28.619950 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:28.620275 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:28.620553 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:28.620555 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:28.620557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:28.620669 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:28.621096 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:28.621100 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:28.621276 (+ 176us) client_negotiation.cc:769] Sending connection context
0809 09:27:28.621397 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 09:27:28.621592 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260809 09:27:28.623375 10251 heartbeater.cc:347] Connected to a master server at 127.9.102.62:35859
I20260809 09:27:28.623585 10251 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:28.623986 10251 heartbeater.cc:511] Master 127.9.102.62:35859 requested a full tablet report, sending...
I20260809 09:27:28.624928 9719 ts_manager.cc:194] Registered new tserver with Master: fb065bca04bd47a7a41cdfc3c1a389f2 (127.9.102.2:41381)
I20260809 09:27:28.625425 9719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:41469
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:27:28.644307 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:35859
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:28.741650 10257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:28.741842 10257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:28.741884 10257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:28.745203 10257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:28.745260 10257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:28.745309 10257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:27:28.749711 10257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:35859
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:28.750415 10257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:28.751163 10257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:28.759644 10271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:28.759660 10274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:28.759725 10270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:28.759716 10257 server_base.cc:1034] running on GCE node
I20260809 09:27:28.760291 10257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:28.760819 10257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:28.762007 10257 hybrid_clock.cc:648] HybridClock initialized: now 1786267648761992 us; error 35 us; skew 500 ppm
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[9633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:28.764627 10257 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:27:28.765457 10257 webserver.cc:460] Webserver started at http://127.9.102.3:37985/ using document root <none> and password file <none>
I20260809 09:27:28.765930 10257 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:28.766009 10257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:28.766198 10257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:28.767951 10257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "06c075d5416c4a85a5bf4594f767fb82"
format_stamp: "Formatted at 2026-08-09 09:27:28 on dist-test-slave-7vjv"
server_key: "5b884d4ee2a4d30aa190932abc29c41c"
server_key_iv: "cf19757acba1525bf472f95e31362e02"
server_key_version: "encryptionkey@0"
I20260809 09:27:28.768461 10257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "06c075d5416c4a85a5bf4594f767fb82"
format_stamp: "Formatted at 2026-08-09 09:27:28 on dist-test-slave-7vjv"
server_key: "5b884d4ee2a4d30aa190932abc29c41c"
server_key_iv: "cf19757acba1525bf472f95e31362e02"
server_key_version: "encryptionkey@0"
I20260809 09:27:28.771858 10257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:27:28.773914 10283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:28.774850 10257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 09:27:28.774967 10257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "06c075d5416c4a85a5bf4594f767fb82"
format_stamp: "Formatted at 2026-08-09 09:27:28 on dist-test-slave-7vjv"
server_key: "5b884d4ee2a4d30aa190932abc29c41c"
server_key_iv: "cf19757acba1525bf472f95e31362e02"
server_key_version: "encryptionkey@0"
I20260809 09:27:28.775054 10257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:28.788786 10257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:28.791085 10257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:28.791253 10257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:28.791746 10257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:28.792479 10257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:28.792541 10257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:28.792594 10257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:28.792629 10257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:28.801177 10257 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:42343
I20260809 09:27:28.801204 10482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:42343 every 8 connection(s)
I20260809 09:27:28.801935 10257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:27:28.809360 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 10257
I20260809 09:27:28.809489 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:27:28.809754 9624 external_mini_cluster.cc:1428] Setting key 71a26764c88ef9208bbab9009603ee36
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:28.816671 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:28.803622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:37701 (local address 127.9.102.62:35859)
0809 09:27:28.803758 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:28.803762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:28.804629 (+ 867us) server_negotiation.cc:407] Connection header received
0809 09:27:28.806120 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:28.806122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:28.806166 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:28.806237 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:28.807378 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.808424 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.810086 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.810297 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.810388 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:28.813363 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:28.813366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:28.813368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:28.813390 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:28.814734 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:28.815260 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:28.815263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:28.815264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:28.815340 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:28.815658 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:28.815660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:28.815661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:28.815822 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:28.815896 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:28.816475 (+ 579us) server_negotiation.cc:299] Negotiation successful
0809 09:27:28.816568 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 09:27:28.817549 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:28.803977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:35859 (local address 127.9.102.3:37701)
0809 09:27:28.804456 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:28.804496 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:28.805920 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:28.806400 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:28.806407 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:28.806763 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:28.807234 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:28.807246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.808568 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.808572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:28.809952 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:28.809966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.810461 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.810467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:28.810737 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:28.811450 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:28.811472 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:28.813219 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:28.814859 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:28.814863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:28.814876 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:28.815158 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:28.815432 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:28.815434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:28.815435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:28.815572 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:28.815916 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:28.815920 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:28.816204 (+ 284us) client_negotiation.cc:769] Sending connection context
0809 09:27:28.816412 (+ 208us) client_negotiation.cc:240] Negotiation successful
0809 09:27:28.816646 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":171,"threads_started":1}
I20260809 09:27:28.818679 10483 heartbeater.cc:347] Connected to a master server at 127.9.102.62:35859
I20260809 09:27:28.818904 10483 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:28.819383 10483 heartbeater.cc:511] Master 127.9.102.62:35859 requested a full tablet report, sending...
I20260809 09:27:28.820935 9719 ts_manager.cc:194] Registered new tserver with Master: 06c075d5416c4a85a5bf4594f767fb82 (127.9.102.3:42343)
I20260809 09:27:28.821512 9719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:37701
I20260809 09:27:28.824532 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[9633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267648, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267648, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.102.1@KRBTEST.COM
I20260809 09:27:28.851984 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:28.843273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.9.102.1:40149)
0809 09:27:28.843665 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:28.843674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:28.843695 (+ 21us) server_negotiation.cc:407] Connection header received
0809 09:27:28.843739 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:28.843743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:28.843886 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:28.844000 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:28.844690 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.845430 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.846205 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.846535 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.846617 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:28.848876 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:28.848884 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:28.848894 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:28.848951 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:28.850414 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:28.850795 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:28.850799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:28.850801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:28.850880 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:28.851110 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:28.851112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:28.851113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:28.851319 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:28.851458 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:28.851639 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 09:27:28.851805 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":203,"threads_started":1}
W20260809 09:27:28.852891 9902 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50610
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[9633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[9633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.1@KRBTEST.COM
I20260809 09:27:28.880028 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:28.872029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.9.102.1:40149)
0809 09:27:28.872174 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:28.872178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:28.872408 (+ 230us) server_negotiation.cc:407] Connection header received
0809 09:27:28.872527 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:28.872529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:28.872570 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:28.872632 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:28.873295 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.873766 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.874546 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:28.874754 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:28.874826 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:28.877141 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:28.877144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:28.877146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:28.877169 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:28.878492 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:28.878876 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:28.878879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:28.878880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:28.878991 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:28.879308 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:28.879311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:28.879312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:28.879471 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:28.879579 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:28.879809 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 09:27:28.879922 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:27:28.881059 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 9812
I20260809 09:27:28.896733 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 10044
I20260809 09:27:28.914115 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 10257
I20260809 09:27:28.922027 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 9653
2026-08-09T09:27:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2241 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[10503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[10503](info): set up 2 sockets
Aug 09 09:27:28 dist-test-slave-7vjv krb5kdc[10503](info): commencing operation
krb5kdc: starting...
W20260809 09:27:30.062472 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:27:30 dist-test-slave-7vjv krb5kdc[10503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:27:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:27:30.168715 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:44893
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:44893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:30.267478 10523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:30.267736 10523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:30.267815 10523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:30.271333 10523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:30.271404 10523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:30.271440 10523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:30.271472 10523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:30.271503 10523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:30.275982 10523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:44893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:44893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:30.276886 10523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:30.277891 10523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:30.285507 10535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:30.285590 10534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:30.285686 10523 server_base.cc:1034] running on GCE node
W20260809 09:27:30.285569 10537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:30.286290 10523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:30.287155 10523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:30.288321 10523 hybrid_clock.cc:648] HybridClock initialized: now 1786267650288294 us; error 37 us; skew 500 ppm
Aug 09 09:27:30 dist-test-slave-7vjv krb5kdc[10503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:30.290979 10523 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:27:30.291774 10523 webserver.cc:460] Webserver started at http://127.9.102.62:33211/ using document root <none> and password file <none>
I20260809 09:27:30.292371 10523 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:30.292439 10523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:30.292675 10523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:30.294523 10523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "c492005c868e41af94e80361cc665371"
format_stamp: "Formatted at 2026-08-09 09:27:30 on dist-test-slave-7vjv"
server_key: "28433ee6d2659e5a95fabfa7a1e11223"
server_key_iv: "be1ab5df1d2ab79aebeb08288f502ed6"
server_key_version: "encryptionkey@0"
I20260809 09:27:30.295053 10523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "c492005c868e41af94e80361cc665371"
format_stamp: "Formatted at 2026-08-09 09:27:30 on dist-test-slave-7vjv"
server_key: "28433ee6d2659e5a95fabfa7a1e11223"
server_key_iv: "be1ab5df1d2ab79aebeb08288f502ed6"
server_key_version: "encryptionkey@0"
I20260809 09:27:30.298584 10523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 09:27:30.300904 10551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:30.302014 10523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:30.302161 10523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "c492005c868e41af94e80361cc665371"
format_stamp: "Formatted at 2026-08-09 09:27:30 on dist-test-slave-7vjv"
server_key: "28433ee6d2659e5a95fabfa7a1e11223"
server_key_iv: "be1ab5df1d2ab79aebeb08288f502ed6"
server_key_version: "encryptionkey@0"
I20260809 09:27:30.302268 10523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:30.316689 10523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:30.319058 10523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:30.319234 10523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:30.325575 10523 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:44893
I20260809 09:27:30.325582 10642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:44893 every 8 connection(s)
I20260809 09:27:30.326337 10523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:30.328707 10644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:30.333324 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371: Bootstrap starting.
I20260809 09:27:30.333854 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 10523
I20260809 09:27:30.333954 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:30.334190 9624 external_mini_cluster.cc:1428] Setting key 026914ccf84fb470bfd0958d8bcb3809
I20260809 09:27:30.335858 10644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:30.336763 10644 log.cc:824] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:30.338841 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371: No bootstrap required, opened a new log
I20260809 09:27:30.343145 10644 raft_consensus.cc:348] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492005c868e41af94e80361cc665371" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44893 } }
I20260809 09:27:30.343385 10644 raft_consensus.cc:374] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:30.343453 10644 raft_consensus.cc:729] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c492005c868e41af94e80361cc665371, State: Initialized, Role: FOLLOWER
I20260809 09:27:30.343900 10644 consensus_queue.cc:260] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492005c868e41af94e80361cc665371" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44893 } }
I20260809 09:27:30.344074 10644 raft_consensus.cc:388] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:30.344177 10644 raft_consensus.cc:482] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:30.344300 10644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 09:27:30 dist-test-slave-7vjv krb5kdc[10503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:30.345280 10644 raft_consensus.cc:504] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492005c868e41af94e80361cc665371" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44893 } }
I20260809 09:27:30.345656 10644 leader_election.cc:302] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c492005c868e41af94e80361cc665371; no voters:
I20260809 09:27:30.345954 10644 leader_election.cc:288] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:30.346063 10655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:30.346310 10655 raft_consensus.cc:686] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [term 1 LEADER]: Becoming Leader. State: Replica: c492005c868e41af94e80361cc665371, State: Running, Role: LEADER
I20260809 09:27:30.346625 10655 consensus_queue.cc:237] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492005c868e41af94e80361cc665371" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44893 } }
I20260809 09:27:30.346999 10644 sys_catalog.cc:564] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:30.347762 10657 sys_catalog.cc:455] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c492005c868e41af94e80361cc665371. Latest consensus state: current_term: 1 leader_uuid: "c492005c868e41af94e80361cc665371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492005c868e41af94e80361cc665371" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44893 } } }
I20260809 09:27:30.347781 10656 sys_catalog.cc:455] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c492005c868e41af94e80361cc665371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c492005c868e41af94e80361cc665371" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44893 } } }
I20260809 09:27:30.347981 10656 sys_catalog.cc:458] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:30.348260 10657 sys_catalog.cc:458] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:30.348335 10673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:30.350418 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:30.335344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.9.102.62:44893)
0809 09:27:30.336318 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:30.336333 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:30.336383 (+ 50us) server_negotiation.cc:407] Connection header received
0809 09:27:30.337249 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:30.337270 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:30.337694 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:30.337987 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:30.339056 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.340410 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.341581 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.342304 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.342448 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:30.345228 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:30.345234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:30.345252 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:30.345275 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:30.347176 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:30.347714 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:30.347721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:30.347729 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:30.347876 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:30.348246 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:30.348251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:30.348253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:30.348776 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:30.348962 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:30.349233 (+ 271us) server_negotiation.cc:299] Negotiation successful
0809 09:27:30.349547 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":119,"threads_started":1}
I20260809 09:27:30.351477 10673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:30.355962 10673 catalog_manager.cc:1347] Generated new cluster ID: 7511a244e8ab444c9f8fa0825b45c9d7
I20260809 09:27:30.356015 10673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:30.362676 10673 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:30.363396 10673 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:30.368974 10673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c492005c868e41af94e80361cc665371: Generated new TSK 0
I20260809 09:27:30.369467 10673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:27:30.398815 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:44893
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:30.520922 10691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:30.521133 10691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:30.521174 10691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:30.524566 10691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:30.524616 10691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:30.524679 10691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:27:30.529104 10691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:44893
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:30.529876 10691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:30.530653 10691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:30.538880 10700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:30.538923 10702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:30.538928 10699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:30.539117 10691 server_base.cc:1034] running on GCE node
I20260809 09:27:30.539486 10691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:30.540041 10691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:30.541272 10691 hybrid_clock.cc:648] HybridClock initialized: now 1786267650541266 us; error 28 us; skew 500 ppm
Aug 09 09:27:30 dist-test-slave-7vjv krb5kdc[10503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:30.544052 10691 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:27:30.544821 10691 webserver.cc:460] Webserver started at http://127.9.102.1:33555/ using document root <none> and password file <none>
I20260809 09:27:30.545313 10691 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:30.545396 10691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:30.545598 10691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:30.547390 10691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "b70468b9bbb4412bb8198d411ce382ec"
format_stamp: "Formatted at 2026-08-09 09:27:30 on dist-test-slave-7vjv"
server_key: "f44cb7bc2e47143c58d0338e89c4ebfa"
server_key_iv: "eb8fbfcfbf497d051a68745838c4b763"
server_key_version: "encryptionkey@0"
I20260809 09:27:30.548005 10691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "b70468b9bbb4412bb8198d411ce382ec"
format_stamp: "Formatted at 2026-08-09 09:27:30 on dist-test-slave-7vjv"
server_key: "f44cb7bc2e47143c58d0338e89c4ebfa"
server_key_iv: "eb8fbfcfbf497d051a68745838c4b763"
server_key_version: "encryptionkey@0"
I20260809 09:27:30.551380 10691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:27:30.553565 10718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:30.554590 10691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:30.554710 10691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "b70468b9bbb4412bb8198d411ce382ec"
format_stamp: "Formatted at 2026-08-09 09:27:30 on dist-test-slave-7vjv"
server_key: "f44cb7bc2e47143c58d0338e89c4ebfa"
server_key_iv: "eb8fbfcfbf497d051a68745838c4b763"
server_key_version: "encryptionkey@0"
I20260809 09:27:30.554801 10691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:30.581959 10691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:30.584712 10691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:30.584884 10691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:30.585361 10691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:30.586041 10691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:30.586104 10691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:30.586164 10691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:30.586203 10691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:30.593900 10691 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:36339
I20260809 09:27:30.593923 10920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:36339 every 8 connection(s)
I20260809 09:27:30.594662 10691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:27:30.594998 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 10691
I20260809 09:27:30.595113 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:27:30.595372 9624 external_mini_cluster.cc:1428] Setting key de669d96046d3e1672fa19a4a3eec1d0
Aug 09 09:27:30 dist-test-slave-7vjv krb5kdc[10503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:27:30.608774 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:30.596327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:53241 (local address 127.9.102.62:44893)
0809 09:27:30.596444 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:30.596448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:30.597398 (+ 950us) server_negotiation.cc:407] Connection header received
0809 09:27:30.598430 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:30.598436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:30.598496 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:30.598605 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:30.600022 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.600513 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.602158 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.602514 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.602629 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:30.605455 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:30.605461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:30.605467 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:30.605503 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:30.607046 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:30.607468 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:30.607471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:30.607472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:30.607578 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:30.607876 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:30.607878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:30.607879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:30.608033 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:30.608152 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:30.608569 (+ 417us) server_negotiation.cc:299] Negotiation successful
0809 09:27:30.608660 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
I20260809 09:27:30.609550 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:30.596726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:44893 (local address 127.9.102.1:53241)
0809 09:27:30.597214 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:30.597280 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:30.598224 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:30.598792 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:30.598802 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:30.599252 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:30.599873 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:30.599895 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.600641 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.600647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:30.602026 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:30.602037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.602798 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.602803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:30.603049 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:30.603609 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:30.603631 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:30.605249 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:30.607139 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:30.607143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:30.607152 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:30.607387 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:30.607667 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:30.607670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:30.607672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:30.607799 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:30.608164 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:30.608168 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:30.608356 (+ 188us) client_negotiation.cc:769] Sending connection context
0809 09:27:30.608529 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 09:27:30.608804 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":126,"threads_started":1}
I20260809 09:27:30.610597 10923 heartbeater.cc:347] Connected to a master server at 127.9.102.62:44893
I20260809 09:27:30.610846 10923 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:30.611289 10923 heartbeater.cc:511] Master 127.9.102.62:44893 requested a full tablet report, sending...
I20260809 09:27:30.612653 10578 ts_manager.cc:194] Registered new tserver with Master: b70468b9bbb4412bb8198d411ce382ec (127.9.102.1:36339)
I20260809 09:27:30.613692 10578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:53241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:27:30.622983 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:44893
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:30.721623 10930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:30.721863 10930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:30.721927 10930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:30.725523 10930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:30.725591 10930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:30.725664 10930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:27:30.730291 10930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:44893
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:30.731164 10930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:30.731963 10930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:30.739868 10940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:30.740016 10930 server_base.cc:1034] running on GCE node
W20260809 09:27:30.739885 10941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:30.739879 10947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:30.740479 10930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:30.741070 10930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:30.742246 10930 hybrid_clock.cc:648] HybridClock initialized: now 1786267650742224 us; error 35 us; skew 500 ppm
Aug 09 09:27:30 dist-test-slave-7vjv krb5kdc[10503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:30.744846 10930 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:27:30.745589 10930 webserver.cc:460] Webserver started at http://127.9.102.2:38325/ using document root <none> and password file <none>
I20260809 09:27:30.746021 10930 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:30.746080 10930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:30.746213 10930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:30.747915 10930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "32cdf0f90014499c91baee635259d61e"
format_stamp: "Formatted at 2026-08-09 09:27:30 on dist-test-slave-7vjv"
server_key: "d36f963f55c56a1145c306c74df88129"
server_key_iv: "166453c1d5424567158cbc6dbb066e53"
server_key_version: "encryptionkey@0"
I20260809 09:27:30.748485 10930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "32cdf0f90014499c91baee635259d61e"
format_stamp: "Formatted at 2026-08-09 09:27:30 on dist-test-slave-7vjv"
server_key: "d36f963f55c56a1145c306c74df88129"
server_key_iv: "166453c1d5424567158cbc6dbb066e53"
server_key_version: "encryptionkey@0"
I20260809 09:27:30.751556 10930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 09:27:30.753566 10957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:30.754608 10930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:30.754712 10930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "32cdf0f90014499c91baee635259d61e"
format_stamp: "Formatted at 2026-08-09 09:27:30 on dist-test-slave-7vjv"
server_key: "d36f963f55c56a1145c306c74df88129"
server_key_iv: "166453c1d5424567158cbc6dbb066e53"
server_key_version: "encryptionkey@0"
I20260809 09:27:30.754798 10930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:30.772980 10930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:30.775620 10930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:30.775738 10930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:30.776279 10930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:30.776947 10930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:30.776989 10930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:30.777067 10930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:30.777132 10930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:30.784626 10930 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:37861
I20260809 09:27:30.784706 11143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:37861 every 8 connection(s)
I20260809 09:27:30.785336 10930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:27:30.788260 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 10930
I20260809 09:27:30.788378 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:27:30.788708 9624 external_mini_cluster.cc:1428] Setting key f945bc157fef403b6fe92ced67d2ab03
Aug 09 09:27:30 dist-test-slave-7vjv krb5kdc[10503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:30.797308 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:30.786742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:50989 (local address 127.9.102.62:44893)
0809 09:27:30.786862 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:30.786866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:30.787525 (+ 659us) server_negotiation.cc:407] Connection header received
0809 09:27:30.788556 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:30.788558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:30.788603 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:30.788687 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:30.789786 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.790265 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.791274 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.791478 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.791583 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:30.793901 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:30.793904 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:30.793906 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:30.793927 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:30.795207 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:30.795741 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:30.795747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:30.795750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:30.795849 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:30.796194 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:30.796199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:30.796203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:30.796451 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:30.796565 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:30.797000 (+ 435us) server_negotiation.cc:299] Negotiation successful
0809 09:27:30.797130 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:27:30.797765 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:30.787007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:44893 (local address 127.9.102.2:50989)
0809 09:27:30.787393 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:30.787427 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:30.788313 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:30.788846 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:30.788865 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:30.789261 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:30.789648 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:30.789660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.790381 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.790388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:30.791148 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:30.791157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.791616 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.791620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:30.791769 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:30.792248 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:30.792262 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:30.793761 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:30.795325 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:30.795332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:30.795348 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:30.795640 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:30.795940 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:30.795942 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:30.795943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:30.796062 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:30.796554 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:30.796558 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:30.796758 (+ 200us) client_negotiation.cc:769] Sending connection context
0809 09:27:30.796897 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 09:27:30.797097 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260809 09:27:30.798704 11144 heartbeater.cc:347] Connected to a master server at 127.9.102.62:44893
I20260809 09:27:30.798924 11144 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
I20260809 09:27:30.799423 11144 heartbeater.cc:511] Master 127.9.102.62:44893 requested a full tablet report, sending...
I20260809 09:27:30.800341 10578 ts_manager.cc:194] Registered new tserver with Master: 32cdf0f90014499c91baee635259d61e (127.9.102.2:37861)
I20260809 09:27:30.800838 10578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:50989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:27:30.814975 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:44893
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:41597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:30.915558 11156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:30.915802 11156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:30.915865 11156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:30.919528 11156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:30.919613 11156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:30.919697 11156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:27:30.924223 11156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:41597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:44893
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:30.924968 11156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:30.925794 11156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:30.933951 11161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:30.934046 11165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:30.933967 11162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:30.934389 11156 server_base.cc:1034] running on GCE node
I20260809 09:27:30.934772 11156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:30.935315 11156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:30.936498 11156 hybrid_clock.cc:648] HybridClock initialized: now 1786267650936463 us; error 44 us; skew 500 ppm
Aug 09 09:27:30 dist-test-slave-7vjv krb5kdc[10503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:30.939088 11156 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:27:30.939888 11156 webserver.cc:460] Webserver started at http://127.9.102.3:45325/ using document root <none> and password file <none>
I20260809 09:27:30.940451 11156 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:30.940541 11156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:30.940757 11156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:30.942534 11156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723"
format_stamp: "Formatted at 2026-08-09 09:27:30 on dist-test-slave-7vjv"
server_key: "a52848dd9812363369b36c0a637dce57"
server_key_iv: "ed85da54894ecca3c237700888a6ca05"
server_key_version: "encryptionkey@0"
I20260809 09:27:30.943053 11156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723"
format_stamp: "Formatted at 2026-08-09 09:27:30 on dist-test-slave-7vjv"
server_key: "a52848dd9812363369b36c0a637dce57"
server_key_iv: "ed85da54894ecca3c237700888a6ca05"
server_key_version: "encryptionkey@0"
I20260809 09:27:30.946400 11156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 09:27:30.948915 11175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:30.950131 11156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:30.950238 11156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723"
format_stamp: "Formatted at 2026-08-09 09:27:30 on dist-test-slave-7vjv"
server_key: "a52848dd9812363369b36c0a637dce57"
server_key_iv: "ed85da54894ecca3c237700888a6ca05"
server_key_version: "encryptionkey@0"
I20260809 09:27:30.950320 11156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:30.963703 11156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:30.966066 11156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:30.966202 11156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:30.966603 11156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:30.967262 11156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:30.967301 11156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:30.967351 11156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:30.967370 11156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:30.975713 11156 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:35021
I20260809 09:27:30.975751 11379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:35021 every 8 connection(s)
I20260809 09:27:30.976457 11156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:27:30.979902 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 11156
I20260809 09:27:30.980033 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:27:30.980299 9624 external_mini_cluster.cc:1428] Setting key 8f0262f7b2381c19439946204957e47d
Aug 09 09:27:30 dist-test-slave-7vjv krb5kdc[10503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:30.989004 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:30.977929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:46963 (local address 127.9.102.62:44893)
0809 09:27:30.978061 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:30.978065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:30.978728 (+ 663us) server_negotiation.cc:407] Connection header received
0809 09:27:30.979496 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:30.979499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:30.979567 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:30.979671 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:30.981071 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.981860 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.983379 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.983669 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.983770 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:30.985921 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:30.985924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:30.985927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:30.985964 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:30.987081 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:30.987553 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:30.987557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:30.987559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:30.987639 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:30.987951 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:30.987955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:30.987957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:30.988194 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:30.988299 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:30.988734 (+ 435us) server_negotiation.cc:299] Negotiation successful
0809 09:27:30.988838 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 09:27:30.989689 11383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:30.978227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:44893 (local address 127.9.102.3:46963)
0809 09:27:30.978602 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:30.978638 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:30.979316 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:30.979829 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:30.979834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:30.980295 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:30.980912 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:30.980933 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.982007 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.982011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:30.983273 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:30.983284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.983814 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.983817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:30.984009 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:30.984481 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:30.984495 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:30.985770 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:30.987192 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:30.987198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:30.987210 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:30.987446 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:30.987727 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:30.987730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:30.987732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:30.987859 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:30.988298 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:30.988305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:30.988513 (+ 208us) client_negotiation.cc:769] Sending connection context
0809 09:27:30.988694 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 09:27:30.988964 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260809 09:27:30.990625 11381 heartbeater.cc:347] Connected to a master server at 127.9.102.62:44893
I20260809 09:27:30.990829 11381 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:30.991264 11381 heartbeater.cc:511] Master 127.9.102.62:44893 requested a full tablet report, sending...
I20260809 09:27:30.992208 10578 ts_manager.cc:194] Registered new tserver with Master: da9cb9f6b9cf4f02ac0f81e7a242c723 (127.9.102.3:35021)
I20260809 09:27:30.992698 10578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:46963
I20260809 09:27:30.994347 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:27:31.003293 10650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:30.997297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.9.102.62:44893)
0809 09:27:30.997400 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:30.997403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:30.997568 (+ 165us) server_negotiation.cc:407] Connection header received
0809 09:27:30.997654 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:30.997667 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:30.997696 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:30.997768 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:30.998323 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.998733 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.999486 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:30.999685 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:30.999762 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.000894 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:31.000897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:31.000898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:31.000913 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:31.001941 (+ 1028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:31.002339 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:31.002341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:31.002342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:31.002412 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:31.002650 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:31.002652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:31.002653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:31.002829 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:31.002927 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:31.003112 (+ 185us) server_negotiation.cc:299] Negotiation successful
0809 09:27:31.003197 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 09:27:31.007395 10578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:27:31.010232 10578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:27:31.023566 11407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.018428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:35021 (local address 127.0.0.1:37368)
0809 09:27:31.019390 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:31.019401 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:31.019515 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:31.019922 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:31.019924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:31.019939 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:31.020004 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:31.020008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.021319 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.021322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:31.022339 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:31.022352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.023082 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.023085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:31.023174 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.023206 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 09:27:31.023323 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 09:27:31.023431 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":1,"thread_start_us":117,"threads_started":1}
I20260809 09:27:31.023566 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.018829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.9.102.3:35021)
0809 09:27:31.019247 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:31.019261 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:31.019493 (+ 232us) server_negotiation.cc:407] Connection header received
0809 09:27:31.019627 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:31.019633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:31.019797 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:31.019940 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:31.020133 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.021049 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.022444 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.022942 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.023071 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.023200 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:31.023352 (+ 152us) server_negotiation.cc:299] Negotiation successful
0809 09:27:31.023432 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":146,"threads_started":1}
I20260809 09:27:31.023712 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.017633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:36339 (local address 127.0.0.1:60650)
0809 09:27:31.018336 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:31.018385 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:31.018506 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:31.019269 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:31.019274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:31.019306 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:31.019489 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:31.019496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.020998 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.021000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:31.022078 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:31.022087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.023275 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.023278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:31.023385 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.023442 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 09:27:31.023506 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 09:27:31.023590 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":70,"spinlock_wait_cycles":4224,"thread_start_us":73,"threads_started":1}
I20260809 09:27:31.023783 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.018261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.9.102.2:37861)
0809 09:27:31.018672 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:31.018686 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:31.018706 (+ 20us) server_negotiation.cc:407] Connection header received
0809 09:27:31.018765 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:31.018769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:31.018943 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:31.019087 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:31.019655 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.020722 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.022337 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.023012 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.023101 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.023216 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:31.023575 (+ 359us) server_negotiation.cc:299] Negotiation successful
0809 09:27:31.023653 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":108,"threads_started":1}
I20260809 09:27:31.024057 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.017768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:37861 (local address 127.0.0.1:57324)
0809 09:27:31.018418 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:31.018430 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:31.018515 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:31.019269 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:31.019274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:31.019307 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:31.019489 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:31.019496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.020863 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.020866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:31.022221 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:31.022231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.023275 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.023278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:31.023408 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.023444 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 09:27:31.023506 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 09:27:31.023939 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":642,"spinlock_wait_cycles":6528,"thread_start_us":98,"threads_started":1}
I20260809 09:27:31.024070 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.017762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.9.102.1:36339)
0809 09:27:31.018074 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:31.018084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:31.018646 (+ 562us) server_negotiation.cc:407] Connection header received
0809 09:27:31.018756 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:31.018761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:31.018947 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:31.019087 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:31.020190 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.020858 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.022204 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.022996 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.023097 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.023216 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:31.023863 (+ 647us) server_negotiation.cc:299] Negotiation successful
0809 09:27:31.023949 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":130,"threads_started":1}
I20260809 09:27:31.026652 11254 tablet_service.cc:1467] Processing CreateTablet for tablet 4daf6e918de646a79b4f68e3383edce3 (DEFAULT_TABLE table=test-table [id=2774741080d1413291a30a67dd6c78a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:31.026705 10806 tablet_service.cc:1467] Processing CreateTablet for tablet 4daf6e918de646a79b4f68e3383edce3 (DEFAULT_TABLE table=test-table [id=2774741080d1413291a30a67dd6c78a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:31.027014 11036 tablet_service.cc:1467] Processing CreateTablet for tablet 4daf6e918de646a79b4f68e3383edce3 (DEFAULT_TABLE table=test-table [id=2774741080d1413291a30a67dd6c78a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:31.027639 11254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4daf6e918de646a79b4f68e3383edce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:31.027702 10806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4daf6e918de646a79b4f68e3383edce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:31.027890 11036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4daf6e918de646a79b4f68e3383edce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:31.033330 11415 tablet_bootstrap.cc:492] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723: Bootstrap starting.
I20260809 09:27:31.034508 11418 tablet_bootstrap.cc:492] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e: Bootstrap starting.
I20260809 09:27:31.035117 11417 tablet_bootstrap.cc:492] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec: Bootstrap starting.
I20260809 09:27:31.036504 11415 tablet_bootstrap.cc:654] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:31.036998 11418 tablet_bootstrap.cc:654] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:31.037132 11417 tablet_bootstrap.cc:654] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:31.037709 11415 log.cc:824] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:31.037709 11418 log.cc:824] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:31.037757 11417 log.cc:824] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:31.039624 11415 tablet_bootstrap.cc:492] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723: No bootstrap required, opened a new log
I20260809 09:27:31.039625 11417 tablet_bootstrap.cc:492] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec: No bootstrap required, opened a new log
I20260809 09:27:31.039625 11418 tablet_bootstrap.cc:492] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e: No bootstrap required, opened a new log
I20260809 09:27:31.039813 11418 ts_tablet_manager.cc:1397] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:27:31.039817 11415 ts_tablet_manager.cc:1397] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260809 09:27:31.039848 11417 ts_tablet_manager.cc:1397] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:27:31.042375 11415 raft_consensus.cc:348] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70468b9bbb4412bb8198d411ce382ec" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36339 } } peers { permanent_uuid: "32cdf0f90014499c91baee635259d61e" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 37861 } } peers { permanent_uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35021 } }
I20260809 09:27:31.042624 11415 raft_consensus.cc:374] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:31.042687 11415 raft_consensus.cc:729] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da9cb9f6b9cf4f02ac0f81e7a242c723, State: Initialized, Role: FOLLOWER
I20260809 09:27:31.043141 11415 consensus_queue.cc:260] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70468b9bbb4412bb8198d411ce382ec" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36339 } } peers { permanent_uuid: "32cdf0f90014499c91baee635259d61e" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 37861 } } peers { permanent_uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35021 } }
I20260809 09:27:31.043452 11418 raft_consensus.cc:348] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70468b9bbb4412bb8198d411ce382ec" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36339 } } peers { permanent_uuid: "32cdf0f90014499c91baee635259d61e" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 37861 } } peers { permanent_uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35021 } }
I20260809 09:27:31.043452 11417 raft_consensus.cc:348] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70468b9bbb4412bb8198d411ce382ec" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36339 } } peers { permanent_uuid: "32cdf0f90014499c91baee635259d61e" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 37861 } } peers { permanent_uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35021 } }
I20260809 09:27:31.043706 11417 raft_consensus.cc:374] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:31.043705 11418 raft_consensus.cc:374] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:31.043736 11381 heartbeater.cc:503] Master 127.9.102.62:44893 was elected leader, sending a full tablet report...
I20260809 09:27:31.043757 11418 raft_consensus.cc:729] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32cdf0f90014499c91baee635259d61e, State: Initialized, Role: FOLLOWER
I20260809 09:27:31.043766 11417 raft_consensus.cc:729] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70468b9bbb4412bb8198d411ce382ec, State: Initialized, Role: FOLLOWER
I20260809 09:27:31.044080 11415 ts_tablet_manager.cc:1428] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260809 09:27:31.044190 11418 consensus_queue.cc:260] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70468b9bbb4412bb8198d411ce382ec" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36339 } } peers { permanent_uuid: "32cdf0f90014499c91baee635259d61e" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 37861 } } peers { permanent_uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35021 } }
I20260809 09:27:31.044190 11417 consensus_queue.cc:260] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70468b9bbb4412bb8198d411ce382ec" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36339 } } peers { permanent_uuid: "32cdf0f90014499c91baee635259d61e" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 37861 } } peers { permanent_uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35021 } }
I20260809 09:27:31.045058 11144 heartbeater.cc:503] Master 127.9.102.62:44893 was elected leader, sending a full tablet report...
I20260809 09:27:31.045049 10923 heartbeater.cc:503] Master 127.9.102.62:44893 was elected leader, sending a full tablet report...
I20260809 09:27:31.045145 11417 ts_tablet_manager.cc:1428] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:27:31.047699 11418 ts_tablet_manager.cc:1428] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260809 09:27:31.095484 11427 raft_consensus.cc:482] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:27:31.095647 11427 raft_consensus.cc:504] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70468b9bbb4412bb8198d411ce382ec" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36339 } } peers { permanent_uuid: "32cdf0f90014499c91baee635259d61e" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 37861 } } peers { permanent_uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35021 } }
W20260809 09:27:31.095770 10926 tablet.cc:2369] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:27:31.096753 11427 leader_election.cc:288] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32cdf0f90014499c91baee635259d61e (127.9.102.2:37861), da9cb9f6b9cf4f02ac0f81e7a242c723 (127.9.102.3:35021)
I20260809 09:27:31.100040 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.096954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:37861 (local address 127.9.102.1:40701)
0809 09:27:31.097133 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:31.097147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:31.097229 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:31.097483 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:31.097486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:31.097502 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:31.097631 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:31.097636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.098248 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.098252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:31.098986 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:31.098995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.099621 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.099625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:31.099732 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.099759 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 09:27:31.099845 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 09:27:31.099925 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260809 09:27:31.100044 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.097044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:40701 (local address 127.9.102.2:37861)
0809 09:27:31.097177 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:31.097183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:31.097197 (+ 14us) server_negotiation.cc:407] Connection header received
0809 09:27:31.097349 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:31.097352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:31.097399 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:31.097474 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:31.097728 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.098149 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.099114 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.099492 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.099624 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.099682 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:31.099863 (+ 181us) server_negotiation.cc:299] Negotiation successful
0809 09:27:31.099938 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 09:27:31.100270 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.097089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:53191 (local address 127.9.102.3:35021)
0809 09:27:31.097211 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:31.097216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:31.097401 (+ 185us) server_negotiation.cc:407] Connection header received
0809 09:27:31.097530 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:31.097539 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:31.097581 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:31.097663 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:31.097893 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.098305 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.099294 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.099699 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.099814 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.099882 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:31.100080 (+ 198us) server_negotiation.cc:299] Negotiation successful
0809 09:27:31.100182 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 09:27:31.100279 11429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.096976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:35021 (local address 127.9.102.1:53191)
0809 09:27:31.097295 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:31.097306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:31.097431 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:31.097676 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:31.097678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:31.097693 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:31.097797 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:31.097802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.098409 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.098413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:31.099173 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:31.099182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.099819 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.099823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:31.099947 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.099964 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 09:27:31.100059 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 09:27:31.100153 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
I20260809 09:27:31.100672 11072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4daf6e918de646a79b4f68e3383edce3" candidate_uuid: "b70468b9bbb4412bb8198d411ce382ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32cdf0f90014499c91baee635259d61e" is_pre_election: true
I20260809 09:27:31.100760 11296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4daf6e918de646a79b4f68e3383edce3" candidate_uuid: "b70468b9bbb4412bb8198d411ce382ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723" is_pre_election: true
I20260809 09:27:31.100930 11072 raft_consensus.cc:2454] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b70468b9bbb4412bb8198d411ce382ec in term 0.
I20260809 09:27:31.100993 11296 raft_consensus.cc:2454] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b70468b9bbb4412bb8198d411ce382ec in term 0.
I20260809 09:27:31.101446 10721 leader_election.cc:302] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32cdf0f90014499c91baee635259d61e, b70468b9bbb4412bb8198d411ce382ec; no voters:
I20260809 09:27:31.101684 11427 raft_consensus.cc:2781] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:27:31.101761 11427 raft_consensus.cc:482] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:27:31.101789 11427 raft_consensus.cc:3037] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:31.102672 11427 raft_consensus.cc:504] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70468b9bbb4412bb8198d411ce382ec" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36339 } } peers { permanent_uuid: "32cdf0f90014499c91baee635259d61e" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 37861 } } peers { permanent_uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35021 } }
I20260809 09:27:31.102972 11427 leader_election.cc:288] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [CANDIDATE]: Term 1 election: Requested vote from peers 32cdf0f90014499c91baee635259d61e (127.9.102.2:37861), da9cb9f6b9cf4f02ac0f81e7a242c723 (127.9.102.3:35021)
I20260809 09:27:31.103240 11072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4daf6e918de646a79b4f68e3383edce3" candidate_uuid: "b70468b9bbb4412bb8198d411ce382ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32cdf0f90014499c91baee635259d61e"
I20260809 09:27:31.103248 11296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4daf6e918de646a79b4f68e3383edce3" candidate_uuid: "b70468b9bbb4412bb8198d411ce382ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723"
I20260809 09:27:31.103334 11296 raft_consensus.cc:3037] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:31.103363 11072 raft_consensus.cc:3037] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:31.104276 11296 raft_consensus.cc:2454] T 4daf6e918de646a79b4f68e3383edce3 P da9cb9f6b9cf4f02ac0f81e7a242c723 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b70468b9bbb4412bb8198d411ce382ec in term 1.
I20260809 09:27:31.104292 11072 raft_consensus.cc:2454] T 4daf6e918de646a79b4f68e3383edce3 P 32cdf0f90014499c91baee635259d61e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b70468b9bbb4412bb8198d411ce382ec in term 1.
I20260809 09:27:31.104545 10724 leader_election.cc:302] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b70468b9bbb4412bb8198d411ce382ec, da9cb9f6b9cf4f02ac0f81e7a242c723; no voters:
I20260809 09:27:31.104719 11427 raft_consensus.cc:2781] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:31.104921 11427 raft_consensus.cc:686] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [term 1 LEADER]: Becoming Leader. State: Replica: b70468b9bbb4412bb8198d411ce382ec, State: Running, Role: LEADER
I20260809 09:27:31.105208 11427 consensus_queue.cc:237] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70468b9bbb4412bb8198d411ce382ec" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36339 } } peers { permanent_uuid: "32cdf0f90014499c91baee635259d61e" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 37861 } } peers { permanent_uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35021 } }
I20260809 09:27:31.107757 10577 catalog_manager.cc:5495] T 4daf6e918de646a79b4f68e3383edce3 P b70468b9bbb4412bb8198d411ce382ec reported cstate change: term changed from 0 to 1, leader changed from <none> to b70468b9bbb4412bb8198d411ce382ec (127.9.102.1). New cstate: current_term: 1 leader_uuid: "b70468b9bbb4412bb8198d411ce382ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b70468b9bbb4412bb8198d411ce382ec" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32cdf0f90014499c91baee635259d61e" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 37861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da9cb9f6b9cf4f02ac0f81e7a242c723" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35021 } health_report { overall_health: UNKNOWN } } }
Aug 09 09:27:31 dist-test-slave-7vjv krb5kdc[10503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.1@KRBTEST.COM
I20260809 09:27:31.121795 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.112951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.9.102.1:36339)
0809 09:27:31.113059 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:31.113065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:31.113268 (+ 203us) server_negotiation.cc:407] Connection header received
0809 09:27:31.113405 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:31.113408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:31.113448 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:31.113520 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:31.114132 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.114562 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.115331 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.115521 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.115622 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.117956 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:31.117965 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:31.117983 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:31.118002 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:31.119364 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:31.119849 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:31.119854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:31.119855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:31.119981 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:31.120422 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:31.120428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:31.120432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:31.120722 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:31.120922 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:31.121453 (+ 531us) server_negotiation.cc:299] Negotiation successful
0809 09:27:31.121614 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Aug 09 09:27:31 dist-test-slave-7vjv krb5kdc[10503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 09:27:31 dist-test-slave-7vjv krb5kdc[10503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267651, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.102.1@KRBTEST.COM
I20260809 09:27:31.165126 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.155626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.9.102.1:36339)
0809 09:27:31.155790 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:31.155795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:31.155935 (+ 140us) server_negotiation.cc:407] Connection header received
0809 09:27:31.156079 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:31.156084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:31.156168 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:31.156245 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:31.157050 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.157578 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.158764 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.159160 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.159250 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.162235 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:31.162238 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:31.162241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:31.162261 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:31.163529 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:31.163927 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:31.163930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:31.163932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:31.164010 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:31.164326 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:31.164330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:31.164331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:31.164481 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:31.164612 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:31.164858 (+ 246us) server_negotiation.cc:299] Negotiation successful
0809 09:27:31.164967 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Aug 09 09:27:31 dist-test-slave-7vjv krb5kdc[10503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 09:27:31 dist-test-slave-7vjv krb5kdc[10503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.1@KRBTEST.COM
I20260809 09:27:31.190234 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:31.182705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.9.102.1:36339)
0809 09:27:31.182813 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:31.182817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:31.182971 (+ 154us) server_negotiation.cc:407] Connection header received
0809 09:27:31.183122 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:31.183125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:31.183172 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:31.183257 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:31.183888 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.184387 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.185167 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:31.185390 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:31.185497 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:31.187574 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:31.187578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:31.187580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:31.187607 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:31.188790 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:31.189175 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:31.189178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:31.189179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:31.189243 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:31.189493 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:31.189496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:31.189497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:31.189645 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:31.189782 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:31.189982 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 09:27:31.190087 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260809 09:27:31.191481 10772 tablet_service.cc:2908] Rejecting scan request for tablet 4daf6e918de646a79b4f68e3383edce3: Uninitialized: clean time has not yet been initialized
I20260809 09:27:31.193497 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 10691
I20260809 09:27:31.207191 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 10930
I20260809 09:27:31.219887 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 11156
I20260809 09:27:31.228726 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 10523
2026-08-09T09:27:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2308 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:27:31 dist-test-slave-7vjv krb5kdc[11463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:27:31 dist-test-slave-7vjv krb5kdc[11463](info): set up 2 sockets
Aug 09 09:27:31 dist-test-slave-7vjv krb5kdc[11463](info): commencing operation
krb5kdc: starting...
W20260809 09:27:32.384919 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:27:32 dist-test-slave-7vjv krb5kdc[11463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:27:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:27:32.494400 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39319
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:32.592767 11485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:32.593014 11485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:32.593081 11485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:32.596378 11485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:32.596451 11485 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 09:27:32.596477 11485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:32.596510 11485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:32.596540 11485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:32.596567 11485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:32.601122 11485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39319
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:32.601919 11485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:32.602702 11485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:32.609972 11499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:32.610077 11485 server_base.cc:1034] running on GCE node
W20260809 09:27:32.609985 11497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:32.609992 11496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:32.610795 11485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:32.611707 11485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:32.612885 11485 hybrid_clock.cc:648] HybridClock initialized: now 1786267652612868 us; error 28 us; skew 500 ppm
Aug 09 09:27:32 dist-test-slave-7vjv krb5kdc[11463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267652, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:32.618865 11485 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:27:32.619632 11485 webserver.cc:460] Webserver started at http://127.9.102.62:40491/ using document root <none> and password file <none>
I20260809 09:27:32.620051 11485 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:32.620158 11485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:32.620316 11485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:32.622128 11485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "c7e4cf44c24343c29b339e5c831da4ca"
format_stamp: "Formatted at 2026-08-09 09:27:32 on dist-test-slave-7vjv"
server_key: "38d101eee01f4192b481998bc7e8c8f5"
server_key_iv: "44c91c13dc673e162aa590fbb32e7b22"
server_key_version: "encryptionkey@0"
I20260809 09:27:32.622619 11485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "c7e4cf44c24343c29b339e5c831da4ca"
format_stamp: "Formatted at 2026-08-09 09:27:32 on dist-test-slave-7vjv"
server_key: "38d101eee01f4192b481998bc7e8c8f5"
server_key_iv: "44c91c13dc673e162aa590fbb32e7b22"
server_key_version: "encryptionkey@0"
I20260809 09:27:32.626000 11485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:32.627948 11512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:32.629027 11485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:32.629184 11485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "c7e4cf44c24343c29b339e5c831da4ca"
format_stamp: "Formatted at 2026-08-09 09:27:32 on dist-test-slave-7vjv"
server_key: "38d101eee01f4192b481998bc7e8c8f5"
server_key_iv: "44c91c13dc673e162aa590fbb32e7b22"
server_key_version: "encryptionkey@0"
I20260809 09:27:32.629262 11485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:32.646628 11485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:32.648850 11485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:32.648975 11485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:32.656395 11485 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:39319
I20260809 09:27:32.656422 11619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:39319 every 8 connection(s)
I20260809 09:27:32.657264 11485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:32.659641 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 11485
I20260809 09:27:32.659679 11620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:32.659732 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:32.659926 9624 external_mini_cluster.cc:1428] Setting key 12fb2bc4ca356bb89eabb3a1edc2e2df
I20260809 09:27:32.666170 11620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca: Bootstrap starting.
Aug 09 09:27:32 dist-test-slave-7vjv krb5kdc[11463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:32.668226 11620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:32.669097 11620 log.cc:824] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:32.671226 11620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca: No bootstrap required, opened a new log
I20260809 09:27:32.673614 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:32.661174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.9.102.62:39319)
0809 09:27:32.661569 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:32.661585 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:32.661625 (+ 40us) server_negotiation.cc:407] Connection header received
0809 09:27:32.662252 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:32.662272 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:32.662635 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:32.662952 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:32.663544 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:32.664917 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:32.665735 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:32.666332 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:32.666471 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:32.668591 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:32.668597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:32.668610 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:32.668638 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:32.671031 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:32.671447 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:32.671452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:32.671457 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:32.671618 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:32.671844 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:32.671847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:32.671849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:32.672271 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:32.672432 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:32.672616 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 09:27:32.672855 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":106,"threads_started":1}
I20260809 09:27:32.674564 11620 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e4cf44c24343c29b339e5c831da4ca" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39319 } }
I20260809 09:27:32.674760 11620 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:32.674826 11620 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e4cf44c24343c29b339e5c831da4ca, State: Initialized, Role: FOLLOWER
I20260809 09:27:32.675246 11620 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e4cf44c24343c29b339e5c831da4ca" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39319 } }
I20260809 09:27:32.675439 11620 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:32.675505 11620 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:32.675598 11620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:32.676442 11620 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e4cf44c24343c29b339e5c831da4ca" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39319 } }
I20260809 09:27:32.676764 11620 leader_election.cc:302] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7e4cf44c24343c29b339e5c831da4ca; no voters:
I20260809 09:27:32.677017 11620 leader_election.cc:288] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:32.677150 11629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:32.677359 11629 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [term 1 LEADER]: Becoming Leader. State: Replica: c7e4cf44c24343c29b339e5c831da4ca, State: Running, Role: LEADER
I20260809 09:27:32.677623 11629 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e4cf44c24343c29b339e5c831da4ca" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39319 } }
I20260809 09:27:32.678174 11620 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:32.679459 11631 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7e4cf44c24343c29b339e5c831da4ca. Latest consensus state: current_term: 1 leader_uuid: "c7e4cf44c24343c29b339e5c831da4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e4cf44c24343c29b339e5c831da4ca" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39319 } } }
I20260809 09:27:32.679453 11630 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7e4cf44c24343c29b339e5c831da4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e4cf44c24343c29b339e5c831da4ca" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39319 } } }
I20260809 09:27:32.679569 11631 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:32.679570 11630 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:32.680018 11643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:32.682703 11643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:32.687516 11643 catalog_manager.cc:1347] Generated new cluster ID: 254abc1bfd0649baafae59993610c8e4
I20260809 09:27:32.687585 11643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:32.717518 11643 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:32.718248 11643 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:32.727917 11643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca: Generated new TSK 0
I20260809 09:27:32.728487 11643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:27:32.756201 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39319
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:32.856122 11662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:32.856355 11662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:32.856416 11662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:32.860306 11662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:32.860378 11662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:32.860445 11662 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 09:27:32.860522 11662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:27:32.864806 11662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.102.62:39319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:32.865608 11662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:32.866474 11662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 09:27:32.875011 11662 server_base.cc:1034] running on GCE node
W20260809 09:27:32.874955 11669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:32.874967 11676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:32.874979 11668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:32.875605 11662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:32.876199 11662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:32.877389 11662 hybrid_clock.cc:648] HybridClock initialized: now 1786267652877383 us; error 26 us; skew 500 ppm
Aug 09 09:27:32 dist-test-slave-7vjv krb5kdc[11463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267652, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:32.880213 11662 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:27:32.880924 11662 webserver.cc:460] Webserver started at http://127.9.102.1:37447/ using document root <none> and password file <none>
I20260809 09:27:32.881349 11662 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:32.881402 11662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:32.881536 11662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:32.883142 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "ef82bc3c24db4178a08f8023d686208f"
format_stamp: "Formatted at 2026-08-09 09:27:32 on dist-test-slave-7vjv"
server_key: "6d30fb81cadf6e75b4fe210c18e452f4"
server_key_iv: "81069854d5c4a3124f65b8703ca64f95"
server_key_version: "encryptionkey@0"
I20260809 09:27:32.883577 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "ef82bc3c24db4178a08f8023d686208f"
format_stamp: "Formatted at 2026-08-09 09:27:32 on dist-test-slave-7vjv"
server_key: "6d30fb81cadf6e75b4fe210c18e452f4"
server_key_iv: "81069854d5c4a3124f65b8703ca64f95"
server_key_version: "encryptionkey@0"
I20260809 09:27:32.886658 11662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:32.888743 11684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:32.889806 11662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:32.889928 11662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "ef82bc3c24db4178a08f8023d686208f"
format_stamp: "Formatted at 2026-08-09 09:27:32 on dist-test-slave-7vjv"
server_key: "6d30fb81cadf6e75b4fe210c18e452f4"
server_key_iv: "81069854d5c4a3124f65b8703ca64f95"
server_key_version: "encryptionkey@0"
I20260809 09:27:32.890024 11662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:32.905678 11662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:32.908066 11662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:32.908267 11662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:32.909474 11662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:32.909538 11662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:32.909597 11662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:32.909643 11662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 09:27:32 dist-test-slave-7vjv krb5kdc[11463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267652, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:32.919021 11662 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:37613
I20260809 09:27:32.919133 11870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:37613 every 8 connection(s)
I20260809 09:27:32.919790 11662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:27:32.921446 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 11662
I20260809 09:27:32.921579 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:27:32.921857 9624 external_mini_cluster.cc:1428] Setting key 471ad1abe0f5445f9ed40b2632ce78de
I20260809 09:27:32.922802 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:32.910020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:58263 (local address 127.9.102.62:39319)
0809 09:27:32.910167 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:32.910173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:32.911345 (+ 1172us) server_negotiation.cc:407] Connection header received
0809 09:27:32.912390 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:32.912396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:32.912459 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:32.912559 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:32.913578 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:32.914056 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:32.915004 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:32.915266 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:32.915370 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:32.919454 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:32.919459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:32.919466 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:32.919488 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:32.920942 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:32.921482 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:32.921485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:32.921486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:32.921582 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:32.921946 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:32.921951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:32.921953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:32.922113 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:32.922182 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:32.922618 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 09:27:32.922703 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 09:27:32.923554 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:32.910753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39319 (local address 127.9.102.1:58263)
0809 09:27:32.911202 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:32.911244 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:32.912176 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:32.912726 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:32.912732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:32.913066 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:32.913431 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:32.913442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:32.914172 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:32.914175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:32.914881 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:32.914889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:32.915412 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:32.915416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:32.915576 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:32.916045 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:32.916065 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:32.919309 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:32.921065 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:32.921071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:32.921086 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:32.921393 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:32.921683 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:32.921685 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:32.921687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:32.921848 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:32.922216 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:32.922221 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:32.922413 (+ 192us) client_negotiation.cc:769] Sending connection context
0809 09:27:32.922595 (+ 182us) client_negotiation.cc:240] Negotiation successful
0809 09:27:32.922850 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":113,"threads_started":1}
I20260809 09:27:32.924769 11871 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39319
I20260809 09:27:32.924932 11871 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:32.925362 11871 heartbeater.cc:511] Master 127.9.102.62:39319 requested a full tablet report, sending...
I20260809 09:27:32.926762 11535 ts_manager.cc:194] Registered new tserver with Master: ef82bc3c24db4178a08f8023d686208f (127.9.102.1:37613)
I20260809 09:27:32.927857 11535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:58263
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:27:32.933544 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:32.926798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39319 (local address 127.9.102.1:55323)
0809 09:27:32.927060 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:32.927074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:32.927152 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:32.927368 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:32.927373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:32.927577 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:32.927706 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:32.927715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:32.928559 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:32.928563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:32.929427 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:32.929440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:32.929851 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:32.929857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:32.929967 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:32.930345 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:32.930362 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:32.930741 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:32.932238 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:32.932241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:32.932243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:32.932467 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:32.932701 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:32.932703 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:32.932705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:32.932783 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:32.933129 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:32.933134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:32.933236 (+ 102us) client_negotiation.cc:769] Sending connection context
0809 09:27:32.933329 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 09:27:32.933419 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
I20260809 09:27:32.933552 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:32.927036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:55323 (local address 127.9.102.62:39319)
0809 09:27:32.927118 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:32.927122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:32.927134 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:27:32.927232 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:32.927235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:32.927267 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:32.927335 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:32.927817 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:32.928421 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:32.929558 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:32.929742 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:32.929812 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:32.930869 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:32.930871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:32.930873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:32.930890 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:32.932136 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:32.932572 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:32.932575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:32.932576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:32.932638 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:32.932855 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:32.932857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:32.932858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:32.933013 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:32.933114 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:32.933382 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 09:27:32.933460 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:27:32.948315 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39319
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:33.048452 11882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:33.048678 11882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:33.048745 11882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:33.052204 11882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:33.052277 11882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:33.052340 11882 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 09:27:33.052412 11882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:27:33.056941 11882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.102.62:39319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:33.057905 11882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:33.058794 11882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:33.067319 11891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:33.067337 11895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:33.067334 11892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:33.067559 11882 server_base.cc:1034] running on GCE node
I20260809 09:27:33.067956 11882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:33.068516 11882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:33.069692 11882 hybrid_clock.cc:648] HybridClock initialized: now 1786267653069669 us; error 36 us; skew 500 ppm
Aug 09 09:27:33 dist-test-slave-7vjv krb5kdc[11463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267653, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:33.072305 11882 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:27:33.073100 11882 webserver.cc:460] Webserver started at http://127.9.102.2:41209/ using document root <none> and password file <none>
I20260809 09:27:33.073580 11882 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:33.073637 11882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:33.073798 11882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:33.075498 11882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "f6d2777a2a3a478ca3ba3faa864e8050"
format_stamp: "Formatted at 2026-08-09 09:27:33 on dist-test-slave-7vjv"
server_key: "b6dd2f1df6536561987af697a76e2afc"
server_key_iv: "f9e673c661c439bc0b0b8d92df5ed1c8"
server_key_version: "encryptionkey@0"
I20260809 09:27:33.075949 11882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "f6d2777a2a3a478ca3ba3faa864e8050"
format_stamp: "Formatted at 2026-08-09 09:27:33 on dist-test-slave-7vjv"
server_key: "b6dd2f1df6536561987af697a76e2afc"
server_key_iv: "f9e673c661c439bc0b0b8d92df5ed1c8"
server_key_version: "encryptionkey@0"
I20260809 09:27:33.079581 11882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 09:27:33.081844 11905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:33.082898 11882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 09:27:33.082984 11882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "f6d2777a2a3a478ca3ba3faa864e8050"
format_stamp: "Formatted at 2026-08-09 09:27:33 on dist-test-slave-7vjv"
server_key: "b6dd2f1df6536561987af697a76e2afc"
server_key_iv: "f9e673c661c439bc0b0b8d92df5ed1c8"
server_key_version: "encryptionkey@0"
I20260809 09:27:33.083058 11882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:33.098555 11882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:33.100978 11882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:33.101104 11882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:33.102312 11882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:33.102358 11882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:33.102418 11882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:33.102452 11882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 09:27:33 dist-test-slave-7vjv krb5kdc[11463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267653, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:33.113018 11882 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:33999
I20260809 09:27:33.113224 12096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:33999 every 8 connection(s)
I20260809 09:27:33.113921 11882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:27:33.115743 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.102908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:41999 (local address 127.9.102.62:39319)
0809 09:27:33.103047 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.103050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.104492 (+ 1442us) server_negotiation.cc:407] Connection header received
0809 09:27:33.105475 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.105478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.105569 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.105657 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:33.106763 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.107253 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.108348 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.108609 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.108728 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.111960 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:33.111964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:33.111966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:33.111990 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:33.113713 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.114286 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.114290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.114291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.114390 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.114734 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.114737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.114739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.114929 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:33.115044 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.115465 (+ 421us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.115585 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:27:33.116436 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.103868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39319 (local address 127.9.102.2:41999)
0809 09:27:33.104338 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:33.104384 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:33.105265 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:33.105807 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:33.105813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:33.106084 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:33.106616 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.106637 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.107366 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.107369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.108236 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.108246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.108800 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.108807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.108981 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.109558 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:33.109582 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:33.111823 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:33.113823 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:33.113828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:33.113840 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:33.114155 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:33.114486 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:33.114489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:33.114491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:33.114651 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:33.115036 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:33.115042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:33.115255 (+ 213us) client_negotiation.cc:769] Sending connection context
0809 09:27:33.115422 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 09:27:33.115694 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":148,"threads_started":1}
I20260809 09:27:33.117430 12098 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39319
I20260809 09:27:33.117583 12098 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:33.118031 12098 heartbeater.cc:511] Master 127.9.102.62:39319 requested a full tablet report, sending...
I20260809 09:27:33.118965 11536 ts_manager.cc:194] Registered new tserver with Master: f6d2777a2a3a478ca3ba3faa864e8050 (127.9.102.2:33999)
I20260809 09:27:33.119756 11536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:41999
I20260809 09:27:33.123831 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 11882
I20260809 09:27:33.123955 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:27:33.124343 9624 external_mini_cluster.cc:1428] Setting key 9cf70537dc794f4bb250dcbd8d4400d6
I20260809 09:27:33.126039 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.119316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39319 (local address 127.9.102.2:51139)
0809 09:27:33.119654 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:33.119667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:33.119755 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:33.120026 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:33.120029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:33.120266 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:33.120392 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.120398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.121277 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.121279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.121811 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.121819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.122186 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.122188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.122248 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.122660 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:33.122673 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:33.122932 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:33.124551 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:33.124554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:33.124555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:33.124793 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:33.125107 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:33.125110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:33.125111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:33.125189 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:33.125566 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:33.125569 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:33.125651 (+ 82us) client_negotiation.cc:769] Sending connection context
0809 09:27:33.125759 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 09:27:33.125877 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260809 09:27:33.126075 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.119381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:51139 (local address 127.9.102.62:39319)
0809 09:27:33.119478 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.119480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.119731 (+ 251us) server_negotiation.cc:407] Connection header received
0809 09:27:33.119884 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.119888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.119933 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.120001 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:33.120507 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.121174 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.121912 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.122097 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.122168 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.123040 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:33.123042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:33.123043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:33.123058 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:33.124407 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.124931 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.124935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.124936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.125022 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.125275 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.125279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.125280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.125465 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:33.125573 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.125811 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.125905 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:27:33.148751 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39319
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:46513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:33.249302 12114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:33.249488 12114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:33.249529 12114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:33.253201 12114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:33.253242 12114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:33.253273 12114 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 09:27:33.253340 12114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:27:33.257952 12114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:46513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.102.62:39319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:33.258765 12114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:33.259655 12114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:33.268337 12122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:33.268340 12126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:33.268332 12123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:33.268612 12114 server_base.cc:1034] running on GCE node
I20260809 09:27:33.268993 12114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:33.269542 12114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:33.270720 12114 hybrid_clock.cc:648] HybridClock initialized: now 1786267653270708 us; error 30 us; skew 500 ppm
Aug 09 09:27:33 dist-test-slave-7vjv krb5kdc[11463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267653, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:33.273264 12114 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:27:33.273974 12114 webserver.cc:460] Webserver started at http://127.9.102.3:34109/ using document root <none> and password file <none>
I20260809 09:27:33.274453 12114 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:33.274541 12114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:33.274746 12114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:33.276721 12114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "6e2d5f66c49d49c585a2a28c58235d3b"
format_stamp: "Formatted at 2026-08-09 09:27:33 on dist-test-slave-7vjv"
server_key: "5f237a58342bbcdd216097fc6ad7e2bf"
server_key_iv: "cc12caa546afe07e94c23f228697afaf"
server_key_version: "encryptionkey@0"
I20260809 09:27:33.277251 12114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "6e2d5f66c49d49c585a2a28c58235d3b"
format_stamp: "Formatted at 2026-08-09 09:27:33 on dist-test-slave-7vjv"
server_key: "5f237a58342bbcdd216097fc6ad7e2bf"
server_key_iv: "cc12caa546afe07e94c23f228697afaf"
server_key_version: "encryptionkey@0"
I20260809 09:27:33.280726 12114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:27:33.282814 12136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:33.283880 12114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:33.284009 12114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "6e2d5f66c49d49c585a2a28c58235d3b"
format_stamp: "Formatted at 2026-08-09 09:27:33 on dist-test-slave-7vjv"
server_key: "5f237a58342bbcdd216097fc6ad7e2bf"
server_key_iv: "cc12caa546afe07e94c23f228697afaf"
server_key_version: "encryptionkey@0"
I20260809 09:27:33.284139 12114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:33.304931 12114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:33.307400 12114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:33.307591 12114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:33.308954 12114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:33.309024 12114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:33.309096 12114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:33.309137 12114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 09:27:33 dist-test-slave-7vjv krb5kdc[11463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267653, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:33.320932 12114 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:33743
I20260809 09:27:33.320953 12342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:33743 every 8 connection(s)
I20260809 09:27:33.321966 12114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:27:33.322048 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.309515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:43517 (local address 127.9.102.62:39319)
0809 09:27:33.309691 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.309699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.310402 (+ 703us) server_negotiation.cc:407] Connection header received
0809 09:27:33.311224 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.311227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.311281 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.311352 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:33.312546 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.313062 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.314319 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.314561 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.314685 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.317938 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:33.317943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:33.317946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:33.317979 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:33.319844 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.320393 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.320410 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.320412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.320531 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.321006 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.321011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.321013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.321222 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:33.321296 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.321827 (+ 531us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.321913 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 09:27:33.322947 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.309836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39319 (local address 127.9.102.3:43517)
0809 09:27:33.310243 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:33.310288 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:33.311028 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:33.311494 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:33.311499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:33.311896 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:33.312403 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.312416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.313185 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.313188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.314188 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.314199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.314717 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.314721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.314877 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.315377 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:33.315397 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:33.317755 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:33.319957 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:33.319962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:33.319971 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:33.320278 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:33.320675 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:33.320681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:33.320685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:33.320878 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:33.321323 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:33.321329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:33.321588 (+ 259us) client_negotiation.cc:769] Sending connection context
0809 09:27:33.321782 (+ 194us) client_negotiation.cc:240] Negotiation successful
0809 09:27:33.322091 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":125,"threads_started":1}
I20260809 09:27:33.323935 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 12114
I20260809 09:27:33.324041 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:27:33.324148 12344 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39319
I20260809 09:27:33.324317 12344 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:33.324376 9624 external_mini_cluster.cc:1428] Setting key 750950721e0196f70b4abdd640fdc895
I20260809 09:27:33.324913 12344 heartbeater.cc:511] Master 127.9.102.62:39319 requested a full tablet report, sending...
I20260809 09:27:33.325919 11535 ts_manager.cc:194] Registered new tserver with Master: 6e2d5f66c49d49c585a2a28c58235d3b (127.9.102.3:33743)
I20260809 09:27:33.326692 11535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:43517
I20260809 09:27:33.327564 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:27:33.333631 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.326412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:34927 (local address 127.9.102.62:39319)
0809 09:27:33.326510 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.326515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.326529 (+ 14us) server_negotiation.cc:407] Connection header received
0809 09:27:33.326560 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.326563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.326598 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.326647 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:33.327188 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.327844 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.329125 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.329339 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.329424 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.330456 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:33.330461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:33.330465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:33.330492 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:33.332002 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.332445 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.332448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.332449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.332543 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.332813 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.332816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.332818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.333020 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:33.333129 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.333406 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.333498 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260809 09:27:33.333640 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.326067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39319 (local address 127.9.102.3:34927)
0809 09:27:33.326299 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:33.326313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:33.326392 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:33.326749 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:33.326751 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:33.326961 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:33.327077 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.327083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.327981 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.327985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.328886 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.328901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.329466 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.329472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.329586 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.329995 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:33.330014 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:33.330310 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:33.332117 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:33.332123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:33.332127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:33.332358 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:33.332632 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:33.332635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:33.332636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:33.332729 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:33.333125 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:33.333128 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:33.333228 (+ 100us) client_negotiation.cc:769] Sending connection context
0809 09:27:33.333339 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 09:27:33.333465 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
Aug 09 09:27:33 dist-test-slave-7vjv krb5kdc[11463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267653, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 09:27:33 dist-test-slave-7vjv krb5kdc[11463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267653, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:33.352787 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.345628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.9.102.62:39319)
0809 09:27:33.345748 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.345751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.345760 (+ 9us) server_negotiation.cc:407] Connection header received
0809 09:27:33.345787 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.345788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.345817 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.345898 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:33.346539 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.346954 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.347705 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.347891 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.347987 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.350246 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:33.350249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:33.350250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:33.350271 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:33.351452 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.351803 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.351805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.351806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.351878 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.352154 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.352156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.352158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.352308 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:33.352423 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.352615 (+ 192us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.352694 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:27:33.355419 11536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:27:33.358553 11536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:27:33.371325 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.366505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:33999 (local address 127.0.0.1:46328)
0809 09:27:33.366863 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:33.366880 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:33.366975 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:33.367689 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:33.367692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:33.367710 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:33.367794 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.367800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.369148 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.369151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.369873 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.369883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.370844 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.370847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.370951 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.370995 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 09:27:33.371096 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 09:27:33.371198 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":21,"thread_start_us":105,"threads_started":1}
I20260809 09:27:33.371448 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.366186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.9.102.3:33743)
0809 09:27:33.366569 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.366580 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.366601 (+ 21us) server_negotiation.cc:407] Connection header received
0809 09:27:33.366784 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.366791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.367117 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.367257 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:33.367653 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.368720 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.370266 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.370952 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.371049 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.371198 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.371252 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.371333 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":115,"threads_started":1}
I20260809 09:27:33.371722 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.366089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:33743 (local address 127.0.0.1:49112)
0809 09:27:33.366417 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:33.366465 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:33.366592 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:33.367395 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:33.367399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:33.367419 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:33.367513 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.367518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.368828 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.368831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.370143 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.370153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.371078 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.371084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.371159 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.371178 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 09:27:33.371231 (+ 53us) client_negotiation.cc:240] Negotiation successful
0809 09:27:33.371280 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":27,"thread_start_us":82,"threads_started":1}
I20260809 09:27:33.372148 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.366855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.9.102.2:33999)
0809 09:27:33.367280 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.367290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.367310 (+ 20us) server_negotiation.cc:407] Connection header received
0809 09:27:33.367382 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.367386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.367550 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.367937 (+ 387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:33.368063 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.369047 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.369991 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.370724 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.370832 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.370981 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.371874 (+ 893us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.371984 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":114,"threads_started":1}
I20260809 09:27:33.372438 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.366849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:37613 (local address 127.0.0.1:38998)
0809 09:27:33.367966 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:33.367977 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:33.368084 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:33.368489 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:33.368492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:33.368510 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:33.368618 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.368626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.370000 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.370008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.371057 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.371067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.372107 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.372111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.372186 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.372204 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 09:27:33.372270 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 09:27:33.372332 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":97,"threads_started":1}
I20260809 09:27:33.372490 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.367395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38998 (local address 127.9.102.1:37613)
0809 09:27:33.367685 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.367699 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.368069 (+ 370us) server_negotiation.cc:407] Connection header received
0809 09:27:33.368215 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.368219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.368361 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.368508 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:33.368708 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.369759 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.371183 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.371984 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.372085 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.372249 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.372297 (+ 48us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.372371 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":115,"threads_started":1}
I20260809 09:27:33.374336 11992 tablet_service.cc:1467] Processing CreateTablet for tablet 3d51f68cc23748ed8c0ebe74a00f3c23 (DEFAULT_TABLE table=test-table [id=bc2cd9c52a5845e6a2a0c60e818b2601]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:33.374362 12225 tablet_service.cc:1467] Processing CreateTablet for tablet 3d51f68cc23748ed8c0ebe74a00f3c23 (DEFAULT_TABLE table=test-table [id=bc2cd9c52a5845e6a2a0c60e818b2601]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:33.375116 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d51f68cc23748ed8c0ebe74a00f3c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:33.375262 12225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d51f68cc23748ed8c0ebe74a00f3c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:33.374951 11769 tablet_service.cc:1467] Processing CreateTablet for tablet 3d51f68cc23748ed8c0ebe74a00f3c23 (DEFAULT_TABLE table=test-table [id=bc2cd9c52a5845e6a2a0c60e818b2601]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:33.375746 11769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d51f68cc23748ed8c0ebe74a00f3c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:33.381368 12380 tablet_bootstrap.cc:492] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050: Bootstrap starting.
I20260809 09:27:33.382745 12382 tablet_bootstrap.cc:492] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b: Bootstrap starting.
I20260809 09:27:33.383347 12384 tablet_bootstrap.cc:492] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f: Bootstrap starting.
I20260809 09:27:33.383631 12380 tablet_bootstrap.cc:654] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:33.384469 12380 log.cc:824] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:33.384727 12382 tablet_bootstrap.cc:654] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:33.385222 12384 tablet_bootstrap.cc:654] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:33.385474 12382 log.cc:824] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:33.385993 12384 log.cc:824] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:33.386477 12380 tablet_bootstrap.cc:492] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050: No bootstrap required, opened a new log
I20260809 09:27:33.386672 12380 ts_tablet_manager.cc:1397] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:27:33.387399 12382 tablet_bootstrap.cc:492] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b: No bootstrap required, opened a new log
I20260809 09:27:33.387581 12382 ts_tablet_manager.cc:1397] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:27:33.388127 12384 tablet_bootstrap.cc:492] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f: No bootstrap required, opened a new log
I20260809 09:27:33.388330 12384 ts_tablet_manager.cc:1397] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260809 09:27:33.389751 12380 raft_consensus.cc:348] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } }
I20260809 09:27:33.390069 12380 raft_consensus.cc:374] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:33.390143 12380 raft_consensus.cc:729] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d2777a2a3a478ca3ba3faa864e8050, State: Initialized, Role: FOLLOWER
I20260809 09:27:33.390563 12380 consensus_queue.cc:260] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } }
I20260809 09:27:33.391041 12382 raft_consensus.cc:348] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } }
I20260809 09:27:33.391263 12382 raft_consensus.cc:374] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:33.391325 12382 raft_consensus.cc:729] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2d5f66c49d49c585a2a28c58235d3b, State: Initialized, Role: FOLLOWER
I20260809 09:27:33.391371 12380 ts_tablet_manager.cc:1428] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 09:27:33.391731 12098 heartbeater.cc:503] Master 127.9.102.62:39319 was elected leader, sending a full tablet report...
I20260809 09:27:33.391701 12382 consensus_queue.cc:260] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } }
I20260809 09:27:33.392174 12384 raft_consensus.cc:348] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } }
I20260809 09:27:33.392329 12344 heartbeater.cc:503] Master 127.9.102.62:39319 was elected leader, sending a full tablet report...
I20260809 09:27:33.392388 12384 raft_consensus.cc:374] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:33.392432 12384 raft_consensus.cc:729] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef82bc3c24db4178a08f8023d686208f, State: Initialized, Role: FOLLOWER
I20260809 09:27:33.392765 12382 ts_tablet_manager.cc:1428] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:27:33.392817 12384 consensus_queue.cc:260] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } }
I20260809 09:27:33.393311 11871 heartbeater.cc:503] Master 127.9.102.62:39319 was elected leader, sending a full tablet report...
I20260809 09:27:33.393668 12384 ts_tablet_manager.cc:1428] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260809 09:27:33.421044 11872 tablet.cc:2369] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:27:33.552268 12392 raft_consensus.cc:482] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:27:33.552493 12392 raft_consensus.cc:504] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } }
I20260809 09:27:33.553632 12392 leader_election.cc:288] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6d2777a2a3a478ca3ba3faa864e8050 (127.9.102.2:33999), ef82bc3c24db4178a08f8023d686208f (127.9.102.1:37613)
I20260809 09:27:33.558197 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.554010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:60213 (local address 127.9.102.2:33999)
0809 09:27:33.554157 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.554161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.554176 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:27:33.554306 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.554309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.554360 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.554429 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:33.554874 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.555545 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.556841 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.557598 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.557722 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.557813 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.558002 (+ 189us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.558088 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 09:27:33.558197 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.553921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:33999 (local address 127.9.102.3:60213)
0809 09:27:33.554056 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:33.554073 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:33.554171 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:33.554606 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:33.554609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:33.554628 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:33.554742 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.554752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.555647 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.555653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.556684 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.556697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.557742 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.557746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.557862 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.557892 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 09:27:33.557980 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 09:27:33.558074 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 09:27:33.559084 12022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51f68cc23748ed8c0ebe74a00f3c23" candidate_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" is_pre_election: true
I20260809 09:27:33.559360 12022 raft_consensus.cc:2454] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e2d5f66c49d49c585a2a28c58235d3b in term 0.
I20260809 09:27:33.559532 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.554360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:41511 (local address 127.9.102.1:37613)
0809 09:27:33.554478 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.554482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.554801 (+ 319us) server_negotiation.cc:407] Connection header received
0809 09:27:33.554894 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.554898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.554960 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.555057 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:33.555339 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.556294 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.557889 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.558657 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.558787 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.558880 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.559274 (+ 394us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.559376 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:27:33.559914 12138 leader_election.cc:302] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e2d5f66c49d49c585a2a28c58235d3b, f6d2777a2a3a478ca3ba3faa864e8050; no voters:
I20260809 09:27:33.559995 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.554242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:37613 (local address 127.9.102.3:41511)
0809 09:27:33.554698 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:33.554713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:33.554807 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:33.555092 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:33.555095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:33.555113 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:33.555220 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.555226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.556409 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.556412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.557729 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.557738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.558956 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.558962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.559111 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.559145 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 09:27:33.559706 (+ 561us) client_negotiation.cc:240] Negotiation successful
0809 09:27:33.559805 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":134,"threads_started":1}
I20260809 09:27:33.560230 12392 raft_consensus.cc:2781] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:27:33.560333 12392 raft_consensus.cc:482] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:27:33.560389 12392 raft_consensus.cc:3037] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:33.560648 11796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51f68cc23748ed8c0ebe74a00f3c23" candidate_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef82bc3c24db4178a08f8023d686208f" is_pre_election: true
I20260809 09:27:33.560914 11796 raft_consensus.cc:2454] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e2d5f66c49d49c585a2a28c58235d3b in term 0.
I20260809 09:27:33.561987 12392 raft_consensus.cc:504] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } }
I20260809 09:27:33.562395 12392 leader_election.cc:288] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [CANDIDATE]: Term 1 election: Requested vote from peers f6d2777a2a3a478ca3ba3faa864e8050 (127.9.102.2:33999), ef82bc3c24db4178a08f8023d686208f (127.9.102.1:37613)
I20260809 09:27:33.562674 12022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51f68cc23748ed8c0ebe74a00f3c23" candidate_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d2777a2a3a478ca3ba3faa864e8050"
I20260809 09:27:33.562798 11796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51f68cc23748ed8c0ebe74a00f3c23" candidate_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef82bc3c24db4178a08f8023d686208f"
I20260809 09:27:33.562816 12022 raft_consensus.cc:3037] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:33.562911 11796 raft_consensus.cc:3037] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:33.564177 11796 raft_consensus.cc:2454] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2d5f66c49d49c585a2a28c58235d3b in term 1.
I20260809 09:27:33.564472 12138 leader_election.cc:302] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e2d5f66c49d49c585a2a28c58235d3b, ef82bc3c24db4178a08f8023d686208f; no voters:
I20260809 09:27:33.564611 12022 raft_consensus.cc:2454] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2d5f66c49d49c585a2a28c58235d3b in term 1.
I20260809 09:27:33.564651 12392 raft_consensus.cc:2781] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:33.564837 12392 raft_consensus.cc:686] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 LEADER]: Becoming Leader. State: Replica: 6e2d5f66c49d49c585a2a28c58235d3b, State: Running, Role: LEADER
I20260809 09:27:33.565165 12392 consensus_queue.cc:237] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } }
I20260809 09:27:33.568521 11536 catalog_manager.cc:5495] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e2d5f66c49d49c585a2a28c58235d3b (127.9.102.3). New cstate: current_term: 1 leader_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } health_report { overall_health: UNKNOWN } } }
W20260809 09:27:33.572880 12346 tablet.cc:2369] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:27:33.610260 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.606910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.9.102.3:33743)
0809 09:27:33.607045 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.607050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.607078 (+ 28us) server_negotiation.cc:407] Connection header received
0809 09:27:33.607141 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.607143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.607187 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.607274 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:33.607534 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.608020 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.608867 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.609082 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.609176 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.609528 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:33.609872 (+ 344us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:33.610004 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.610092 (+ 88us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.610140 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260809 09:27:33.614991 12099 tablet.cc:2369] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:27:33.615255 12022 raft_consensus.cc:1261] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050 [term 1 FOLLOWER]: Refusing update from remote peer 6e2d5f66c49d49c585a2a28c58235d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:33.615278 11796 raft_consensus.cc:1261] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f [term 1 FOLLOWER]: Refusing update from remote peer 6e2d5f66c49d49c585a2a28c58235d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:33.615927 12401 consensus_queue.cc:1035] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:33.616113 12392 consensus_queue.cc:1035] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:33.622956 12407 mvcc.cc:205] Tried to move back new op lower bound from 7316552309201674240 to 7316552309005885440. Current Snapshot: MvccSnapshot[applied={T|T < 7316552309201674240}]
I20260809 09:27:33.624717 12411 mvcc.cc:205] Tried to move back new op lower bound from 7316552309201674240 to 7316552309005885440. Current Snapshot: MvccSnapshot[applied={T|T < 7316552309201674240}]
I20260809 09:27:33.628039 12410 mvcc.cc:205] Tried to move back new op lower bound from 7316552309201674240 to 7316552309005885440. Current Snapshot: MvccSnapshot[applied={T|T < 7316552309201674240}]
I20260809 09:27:33.635653 11536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:33.635888 11536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:33.638291 11536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca: Sending DeleteTablet for 3 replicas of tablet 3d51f68cc23748ed8c0ebe74a00f3c23
I20260809 09:27:33.639000 12225 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d51f68cc23748ed8c0ebe74a00f3c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:33 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:49112
I20260809 09:27:33.639175 11992 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d51f68cc23748ed8c0ebe74a00f3c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:33 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:46328
I20260809 09:27:33.639623 12422 tablet_replica.cc:330] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b: stopping tablet replica
I20260809 09:27:33.639654 11769 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d51f68cc23748ed8c0ebe74a00f3c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:33 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:38998
I20260809 09:27:33.639804 11536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 09:27:33.639921 12422 raft_consensus.cc:2229] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 LEADER]: Raft consensus shutting down.
W20260809 09:27:33.640185 11536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:27:33.640280 12422 raft_consensus.cc:2258] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:33.640548 12424 tablet_replica.cc:330] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050: stopping tablet replica
I20260809 09:27:33.640830 12425 tablet_replica.cc:330] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f: stopping tablet replica
I20260809 09:27:33.641831 12422 ts_tablet_manager.cc:1905] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:33.642433 12425 raft_consensus.cc:2229] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:33.642701 12425 raft_consensus.cc:2258] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:33.641664 12424 raft_consensus.cc:2229] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:33.644183 12424 raft_consensus.cc:2258] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:33.644773 11992 tablet_service.cc:1467] Processing CreateTablet for tablet 58b1b58f1d57433c9175a881614cea4b (DEFAULT_TABLE table=test-table [id=56593fd2e1574b7ab292270b6e619b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:33.644908 11769 tablet_service.cc:1467] Processing CreateTablet for tablet 58b1b58f1d57433c9175a881614cea4b (DEFAULT_TABLE table=test-table [id=56593fd2e1574b7ab292270b6e619b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:33.645038 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58b1b58f1d57433c9175a881614cea4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:33.645174 11769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58b1b58f1d57433c9175a881614cea4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:33.646987 12384 tablet_bootstrap.cc:492] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f: Bootstrap starting.
I20260809 09:27:33.647413 12380 tablet_bootstrap.cc:492] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050: Bootstrap starting.
I20260809 09:27:33.647502 12225 tablet_service.cc:1467] Processing CreateTablet for tablet 58b1b58f1d57433c9175a881614cea4b (DEFAULT_TABLE table=test-table [id=56593fd2e1574b7ab292270b6e619b1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:33.647715 12225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58b1b58f1d57433c9175a881614cea4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:33.647898 12384 tablet_bootstrap.cc:654] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:33.648440 12380 tablet_bootstrap.cc:654] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:33.649784 12382 tablet_bootstrap.cc:492] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b: Bootstrap starting.
I20260809 09:27:33.649904 12424 ts_tablet_manager.cc:1905] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:33.650751 12382 tablet_bootstrap.cc:654] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:33.651958 12382 tablet_bootstrap.cc:492] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b: No bootstrap required, opened a new log
I20260809 09:27:33.651994 12422 ts_tablet_manager.cc:1918] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:27:33.652037 12382 ts_tablet_manager.cc:1397] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:33.652082 12422 log.cc:1196] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/3d51f68cc23748ed8c0ebe74a00f3c23
I20260809 09:27:33.652329 12424 ts_tablet_manager.cc:1918] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:27:33.652410 12422 ts_tablet_manager.cc:1939] T 3d51f68cc23748ed8c0ebe74a00f3c23 P 6e2d5f66c49d49c585a2a28c58235d3b: Deleting consensus metadata
I20260809 09:27:33.652433 12424 log.cc:1196] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/3d51f68cc23748ed8c0ebe74a00f3c23
I20260809 09:27:33.652738 12424 ts_tablet_manager.cc:1939] T 3d51f68cc23748ed8c0ebe74a00f3c23 P f6d2777a2a3a478ca3ba3faa864e8050: Deleting consensus metadata
I20260809 09:27:33.652697 12382 raft_consensus.cc:348] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.652791 12382 raft_consensus.cc:374] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:33.652829 12382 raft_consensus.cc:729] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2d5f66c49d49c585a2a28c58235d3b, State: Initialized, Role: FOLLOWER
I20260809 09:27:33.652938 12382 consensus_queue.cc:260] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.653184 12382 ts_tablet_manager.cc:1428] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:27:33.654639 12425 ts_tablet_manager.cc:1905] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:33.654738 11517 catalog_manager.cc:4841] TS 6e2d5f66c49d49c585a2a28c58235d3b (127.9.102.3:33743): tablet 3d51f68cc23748ed8c0ebe74a00f3c23 (table test-table [id=bc2cd9c52a5845e6a2a0c60e818b2601]) successfully deleted
I20260809 09:27:33.655233 12384 tablet_bootstrap.cc:492] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f: No bootstrap required, opened a new log
I20260809 09:27:33.655330 12384 ts_tablet_manager.cc:1397] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260809 09:27:33.655696 12384 raft_consensus.cc:348] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.655798 12384 raft_consensus.cc:374] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:33.655829 12384 raft_consensus.cc:729] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef82bc3c24db4178a08f8023d686208f, State: Initialized, Role: FOLLOWER
I20260809 09:27:33.655936 12384 consensus_queue.cc:260] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.656206 12384 ts_tablet_manager.cc:1428] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 09:27:33.656709 11513 catalog_manager.cc:4841] TS f6d2777a2a3a478ca3ba3faa864e8050 (127.9.102.2:33999): tablet 3d51f68cc23748ed8c0ebe74a00f3c23 (table test-table [id=bc2cd9c52a5845e6a2a0c60e818b2601]) successfully deleted
I20260809 09:27:33.657056 12425 ts_tablet_manager.cc:1918] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:27:33.657135 12425 log.cc:1196] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/wal/wals/3d51f68cc23748ed8c0ebe74a00f3c23
I20260809 09:27:33.657425 12425 ts_tablet_manager.cc:1939] T 3d51f68cc23748ed8c0ebe74a00f3c23 P ef82bc3c24db4178a08f8023d686208f: Deleting consensus metadata
I20260809 09:27:33.658385 11513 catalog_manager.cc:4841] TS ef82bc3c24db4178a08f8023d686208f (127.9.102.1:37613): tablet 3d51f68cc23748ed8c0ebe74a00f3c23 (table test-table [id=bc2cd9c52a5845e6a2a0c60e818b2601]) successfully deleted
I20260809 09:27:33.658946 12380 tablet_bootstrap.cc:492] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050: No bootstrap required, opened a new log
I20260809 09:27:33.659015 12380 ts_tablet_manager.cc:1397] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260809 09:27:33.659390 12380 raft_consensus.cc:348] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.659479 12380 raft_consensus.cc:374] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:33.659507 12380 raft_consensus.cc:729] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d2777a2a3a478ca3ba3faa864e8050, State: Initialized, Role: FOLLOWER
I20260809 09:27:33.659615 12380 consensus_queue.cc:260] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.659826 12380 ts_tablet_manager.cc:1428] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:27:33.760066 12392 raft_consensus.cc:482] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:27:33.760195 12392 raft_consensus.cc:504] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.760592 12392 leader_election.cc:288] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef82bc3c24db4178a08f8023d686208f (127.9.102.1:37613), f6d2777a2a3a478ca3ba3faa864e8050 (127.9.102.2:33999)
I20260809 09:27:33.760991 11796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b1b58f1d57433c9175a881614cea4b" candidate_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef82bc3c24db4178a08f8023d686208f" is_pre_election: true
I20260809 09:27:33.761051 12022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b1b58f1d57433c9175a881614cea4b" candidate_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" is_pre_election: true
I20260809 09:27:33.761137 11796 raft_consensus.cc:2454] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e2d5f66c49d49c585a2a28c58235d3b in term 0.
I20260809 09:27:33.761210 12022 raft_consensus.cc:2454] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e2d5f66c49d49c585a2a28c58235d3b in term 0.
I20260809 09:27:33.761543 12138 leader_election.cc:302] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e2d5f66c49d49c585a2a28c58235d3b, f6d2777a2a3a478ca3ba3faa864e8050; no voters:
I20260809 09:27:33.761744 12392 raft_consensus.cc:2781] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:27:33.761799 12392 raft_consensus.cc:482] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:27:33.761842 12392 raft_consensus.cc:3037] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:33.762768 12392 raft_consensus.cc:504] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.763085 12392 leader_election.cc:288] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [CANDIDATE]: Term 1 election: Requested vote from peers ef82bc3c24db4178a08f8023d686208f (127.9.102.1:37613), f6d2777a2a3a478ca3ba3faa864e8050 (127.9.102.2:33999)
I20260809 09:27:33.763393 11796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b1b58f1d57433c9175a881614cea4b" candidate_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef82bc3c24db4178a08f8023d686208f"
I20260809 09:27:33.763411 12022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58b1b58f1d57433c9175a881614cea4b" candidate_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d2777a2a3a478ca3ba3faa864e8050"
I20260809 09:27:33.763500 11796 raft_consensus.cc:3037] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:33.763509 12022 raft_consensus.cc:3037] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:33.764513 11796 raft_consensus.cc:2454] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2d5f66c49d49c585a2a28c58235d3b in term 1.
I20260809 09:27:33.764514 12022 raft_consensus.cc:2454] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2d5f66c49d49c585a2a28c58235d3b in term 1.
I20260809 09:27:33.764794 12138 leader_election.cc:302] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e2d5f66c49d49c585a2a28c58235d3b, ef82bc3c24db4178a08f8023d686208f; no voters:
I20260809 09:27:33.765074 12392 raft_consensus.cc:2781] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:33.765164 12392 raft_consensus.cc:686] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 LEADER]: Becoming Leader. State: Replica: 6e2d5f66c49d49c585a2a28c58235d3b, State: Running, Role: LEADER
I20260809 09:27:33.765303 12392 consensus_queue.cc:237] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.767041 11535 catalog_manager.cc:5495] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e2d5f66c49d49c585a2a28c58235d3b (127.9.102.3). New cstate: current_term: 1 leader_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } health_report { overall_health: HEALTHY } } }
Aug 09 09:27:33 dist-test-slave-7vjv krb5kdc[11463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267652, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:33.829370 12439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.821626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39319 (local address 127.0.0.1:38818)
0809 09:27:33.821930 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:33.821941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:33.822064 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:33.822314 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:33.822317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:33.822738 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:33.822820 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.822827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.823536 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.823538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.824201 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.824211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.824601 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.824603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.824662 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.825197 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:33.825207 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:33.826558 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:33.828133 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:33.828142 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:33.828151 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:33.828368 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:33.828597 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:33.828599 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:33.828611 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:33.828678 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:33.828962 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:33.828966 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:33.829055 (+ 89us) client_negotiation.cc:769] Sending connection context
0809 09:27:33.829135 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 09:27:33.829238 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":92,"threads_started":1}
I20260809 09:27:33.829392 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.821720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.9.102.62:39319)
0809 09:27:33.821847 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.821852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.822021 (+ 169us) server_negotiation.cc:407] Connection header received
0809 09:27:33.822167 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.822170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.822214 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.822298 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:33.822936 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.823426 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.824309 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.824508 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.824586 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.826681 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:33.826685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:33.826686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:33.826705 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:33.828002 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.828448 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.828451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.828452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.828520 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.828746 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.828748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.828749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.828885 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:33.828949 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.829186 (+ 237us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.829286 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":45}
I20260809 09:27:33.832758 11535 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:38818:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 09:27:33.833372 11535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:27:33.837836 11769 tablet_service.cc:1467] Processing CreateTablet for tablet f2927590b5c44c09b2d1fdb38ab27a1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f119a47041144fa8ac987244ccca66e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 09:27:33.838070 12225 tablet_service.cc:1467] Processing CreateTablet for tablet f2927590b5c44c09b2d1fdb38ab27a1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f119a47041144fa8ac987244ccca66e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 09:27:33.838218 11769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2927590b5c44c09b2d1fdb38ab27a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:33.838104 11992 tablet_service.cc:1467] Processing CreateTablet for tablet f2927590b5c44c09b2d1fdb38ab27a1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f119a47041144fa8ac987244ccca66e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 09:27:33.838370 12225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2927590b5c44c09b2d1fdb38ab27a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:33.838394 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2927590b5c44c09b2d1fdb38ab27a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:33.840715 12380 tablet_bootstrap.cc:492] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050: Bootstrap starting.
I20260809 09:27:33.840867 12382 tablet_bootstrap.cc:492] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: Bootstrap starting.
I20260809 09:27:33.840893 12384 tablet_bootstrap.cc:492] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f: Bootstrap starting.
I20260809 09:27:33.841502 12380 tablet_bootstrap.cc:654] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:33.841702 12382 tablet_bootstrap.cc:654] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:33.841701 12384 tablet_bootstrap.cc:654] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:33.842540 12380 tablet_bootstrap.cc:492] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050: No bootstrap required, opened a new log
I20260809 09:27:33.842634 12380 ts_tablet_manager.cc:1397] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:33.842849 12384 tablet_bootstrap.cc:492] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f: No bootstrap required, opened a new log
I20260809 09:27:33.842944 12384 ts_tablet_manager.cc:1397] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260809 09:27:33.843061 12382 tablet_bootstrap.cc:492] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: No bootstrap required, opened a new log
I20260809 09:27:33.843056 12380 raft_consensus.cc:348] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.843138 12382 ts_tablet_manager.cc:1397] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:33.843164 12380 raft_consensus.cc:374] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:33.843200 12380 raft_consensus.cc:729] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d2777a2a3a478ca3ba3faa864e8050, State: Initialized, Role: FOLLOWER
I20260809 09:27:33.843310 12380 consensus_queue.cc:260] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.843382 12384 raft_consensus.cc:348] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.843502 12384 raft_consensus.cc:374] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:33.843536 12384 raft_consensus.cc:729] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef82bc3c24db4178a08f8023d686208f, State: Initialized, Role: FOLLOWER
I20260809 09:27:33.843552 12380 ts_tablet_manager.cc:1428] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 09:27:33.843490 12382 raft_consensus.cc:348] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.843592 12382 raft_consensus.cc:374] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:33.843622 12382 raft_consensus.cc:729] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2d5f66c49d49c585a2a28c58235d3b, State: Initialized, Role: FOLLOWER
I20260809 09:27:33.843644 12384 consensus_queue.cc:260] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.843734 12382 consensus_queue.cc:260] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.843888 12384 ts_tablet_manager.cc:1428] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260809 09:27:33.843967 12382 ts_tablet_manager.cc:1428] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:27:33.844213 12391 tablet_replica.cc:439] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } }
I20260809 09:27:33.844218 12392 tablet_replica.cc:439] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } }
I20260809 09:27:33.844342 12391 tablet_replica.cc:442] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:33.844343 12392 tablet_replica.cc:442] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:33.844553 12395 tablet_replica.cc:439] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } }
I20260809 09:27:33.844682 12395 tablet_replica.cc:442] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:33.887984 12392 raft_consensus.cc:482] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:27:33.888065 12392 raft_consensus.cc:504] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.888388 12392 leader_election.cc:288] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef82bc3c24db4178a08f8023d686208f (127.9.102.1:37613), f6d2777a2a3a478ca3ba3faa864e8050 (127.9.102.2:33999)
I20260809 09:27:33.888739 11796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2927590b5c44c09b2d1fdb38ab27a1f" candidate_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef82bc3c24db4178a08f8023d686208f" is_pre_election: true
I20260809 09:27:33.888835 12022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2927590b5c44c09b2d1fdb38ab27a1f" candidate_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" is_pre_election: true
I20260809 09:27:33.888882 11796 raft_consensus.cc:2454] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e2d5f66c49d49c585a2a28c58235d3b in term 0.
I20260809 09:27:33.888963 12022 raft_consensus.cc:2454] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e2d5f66c49d49c585a2a28c58235d3b in term 0.
I20260809 09:27:33.889185 12138 leader_election.cc:302] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e2d5f66c49d49c585a2a28c58235d3b, ef82bc3c24db4178a08f8023d686208f; no voters:
I20260809 09:27:33.889369 12392 raft_consensus.cc:2781] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:27:33.889436 12392 raft_consensus.cc:482] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:27:33.889484 12392 raft_consensus.cc:3037] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:33.890259 12392 raft_consensus.cc:504] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.890545 12392 leader_election.cc:288] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [CANDIDATE]: Term 1 election: Requested vote from peers ef82bc3c24db4178a08f8023d686208f (127.9.102.1:37613), f6d2777a2a3a478ca3ba3faa864e8050 (127.9.102.2:33999)
I20260809 09:27:33.890851 11796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2927590b5c44c09b2d1fdb38ab27a1f" candidate_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef82bc3c24db4178a08f8023d686208f"
I20260809 09:27:33.890906 12022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2927590b5c44c09b2d1fdb38ab27a1f" candidate_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d2777a2a3a478ca3ba3faa864e8050"
I20260809 09:27:33.890971 11796 raft_consensus.cc:3037] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:33.891004 12022 raft_consensus.cc:3037] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:33.891750 11796 raft_consensus.cc:2454] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2d5f66c49d49c585a2a28c58235d3b in term 1.
I20260809 09:27:33.891767 12022 raft_consensus.cc:2454] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2d5f66c49d49c585a2a28c58235d3b in term 1.
I20260809 09:27:33.892047 12138 leader_election.cc:302] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e2d5f66c49d49c585a2a28c58235d3b, ef82bc3c24db4178a08f8023d686208f; no voters:
I20260809 09:27:33.892333 12392 raft_consensus.cc:2781] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:33.892443 12392 raft_consensus.cc:686] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 LEADER]: Becoming Leader. State: Replica: 6e2d5f66c49d49c585a2a28c58235d3b, State: Running, Role: LEADER
I20260809 09:27:33.892583 12392 consensus_queue.cc:237] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } }
I20260809 09:27:33.893221 12401 tablet_replica.cc:439] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: TxnStatusTablet state changed. Reason: New leader 6e2d5f66c49d49c585a2a28c58235d3b. Latest consensus state: current_term: 1 leader_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } }
I20260809 09:27:33.893342 12401 tablet_replica.cc:442] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: This TxnStatusTablet replica's current role is: LEADER
I20260809 09:27:33.893652 12449 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 09:27:33.893836 12449 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 09:27:33.894152 11534 catalog_manager.cc:5495] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e2d5f66c49d49c585a2a28c58235d3b (127.9.102.3). New cstate: current_term: 1 leader_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } health_report { overall_health: HEALTHY } } }
I20260809 09:27:33.910048 12439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.906801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:33743 (local address 127.0.0.1:49128)
0809 09:27:33.906963 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:33.906978 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:33.907078 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:33.907414 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:33.907416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:33.907426 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 09:27:33.907538 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.907543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.908302 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.908307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.908891 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.908899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.909363 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.909367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.909452 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.909492 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 09:27:33.909789 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 09:27:33.909809 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 09:27:33.909870 (+ 61us) client_negotiation.cc:240] Negotiation successful
0809 09:27:33.909949 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 09:27:33.910043 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.906886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.9.102.3:33743)
0809 09:27:33.907218 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.907222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.907231 (+ 9us) server_negotiation.cc:407] Connection header received
0809 09:27:33.907266 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.907268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.907303 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.907387 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:33.907652 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.908186 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.909014 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.909245 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.909361 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.909632 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:33.909707 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:33.909781 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.909879 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.909923 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268}
I20260809 09:27:33.912238 11796 raft_consensus.cc:1261] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f [term 1 FOLLOWER]: Refusing update from remote peer 6e2d5f66c49d49c585a2a28c58235d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:33.912286 12022 raft_consensus.cc:1261] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050 [term 1 FOLLOWER]: Refusing update from remote peer 6e2d5f66c49d49c585a2a28c58235d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:33.912617 12401 consensus_queue.cc:1035] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:33.912736 12392 consensus_queue.cc:1035] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:33.913972 12395 tablet_replica.cc:439] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f: TxnStatusTablet state changed. Reason: New leader 6e2d5f66c49d49c585a2a28c58235d3b. Latest consensus state: current_term: 1 leader_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } }
I20260809 09:27:33.914076 12395 tablet_replica.cc:442] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:33.914901 12391 tablet_replica.cc:439] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050: TxnStatusTablet state changed. Reason: New leader 6e2d5f66c49d49c585a2a28c58235d3b. Latest consensus state: current_term: 1 leader_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } }
I20260809 09:27:33.915014 12391 tablet_replica.cc:442] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:33.915866 12401 tablet_replica.cc:439] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } }
I20260809 09:27:33.915949 12401 tablet_replica.cc:442] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: This TxnStatusTablet replica's current role is: LEADER
I20260809 09:27:33.916911 12392 tablet_replica.cc:439] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } }
I20260809 09:27:33.917011 12392 tablet_replica.cc:442] T f2927590b5c44c09b2d1fdb38ab27a1f P 6e2d5f66c49d49c585a2a28c58235d3b: This TxnStatusTablet replica's current role is: LEADER
I20260809 09:27:33.917106 12391 tablet_replica.cc:439] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } }
I20260809 09:27:33.917201 12391 tablet_replica.cc:442] T f2927590b5c44c09b2d1fdb38ab27a1f P f6d2777a2a3a478ca3ba3faa864e8050: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:33.918107 12395 tablet_replica.cc:439] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 } } peers { permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 } } peers { permanent_uuid: "6e2d5f66c49d49c585a2a28c58235d3b" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33743 } } }
I20260809 09:27:33.918210 12395 tablet_replica.cc:442] T f2927590b5c44c09b2d1fdb38ab27a1f P ef82bc3c24db4178a08f8023d686208f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:33.931315 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.926565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:55427 (local address 127.9.102.3:33743)
0809 09:27:33.926667 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.926673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.928359 (+ 1686us) server_negotiation.cc:407] Connection header received
0809 09:27:33.928467 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.928470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.928505 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.928579 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:33.928822 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.929397 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.930186 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.930403 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.930518 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.930856 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:33.930921 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:33.930996 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.931078 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.931191 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":33}
I20260809 09:27:33.931322 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.926551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:33743 (local address 127.9.102.3:55427)
0809 09:27:33.928278 (+ 1727us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:33.928289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:33.928398 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:33.928588 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:33.928590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:33.928596 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 09:27:33.928707 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.928712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.929493 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.929499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.930059 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:33.930066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.930519 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.930530 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:33.930685 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.930722 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 09:27:33.930998 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 09:27:33.931006 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 09:27:33.931072 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 09:27:33.931219 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1649,"mutex_wait_us":93,"thread_start_us":105,"threads_started":1}
I20260809 09:27:33.938325 12022 raft_consensus.cc:1261] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050 [term 1 FOLLOWER]: Refusing update from remote peer 6e2d5f66c49d49c585a2a28c58235d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:33.938436 11796 raft_consensus.cc:1261] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f [term 1 FOLLOWER]: Refusing update from remote peer 6e2d5f66c49d49c585a2a28c58235d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:33.938767 12401 consensus_queue.cc:1035] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef82bc3c24db4178a08f8023d686208f" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:33.938925 12392 consensus_queue.cc:1035] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d2777a2a3a478ca3ba3faa864e8050" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:33.972970 11534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:33.973109 11534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:33.974726 11534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7e4cf44c24343c29b339e5c831da4ca: Sending DeleteTablet for 3 replicas of tablet 58b1b58f1d57433c9175a881614cea4b
I20260809 09:27:33.975311 11992 tablet_service.cc:1514] Processing DeleteTablet for tablet 58b1b58f1d57433c9175a881614cea4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:33 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:46328
I20260809 09:27:33.975378 11769 tablet_service.cc:1514] Processing DeleteTablet for tablet 58b1b58f1d57433c9175a881614cea4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:33 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:38998
I20260809 09:27:33.975484 12424 tablet_replica.cc:330] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050: stopping tablet replica
I20260809 09:27:33.975488 12225 tablet_service.cc:1514] Processing DeleteTablet for tablet 58b1b58f1d57433c9175a881614cea4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:33 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:49112
I20260809 09:27:33.975629 12424 raft_consensus.cc:2229] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:33.975658 12422 tablet_replica.cc:330] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b: stopping tablet replica
I20260809 09:27:33.975746 12424 raft_consensus.cc:2258] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:33.975757 12422 raft_consensus.cc:2229] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:27:33.975894 12422 raft_consensus.cc:2258] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:33.975920 12425 tablet_replica.cc:330] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f: stopping tablet replica
I20260809 09:27:33.976037 12425 raft_consensus.cc:2229] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:33.976176 12425 raft_consensus.cc:2258] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:33.976609 12422 ts_tablet_manager.cc:1905] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:33.976934 12424 ts_tablet_manager.cc:1905] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:33.979123 12422 ts_tablet_manager.cc:1918] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:27:33.979210 12422 log.cc:1196] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/58b1b58f1d57433c9175a881614cea4b
I20260809 09:27:33.979461 12422 ts_tablet_manager.cc:1939] T 58b1b58f1d57433c9175a881614cea4b P 6e2d5f66c49d49c585a2a28c58235d3b: Deleting consensus metadata
I20260809 09:27:33.979545 12424 ts_tablet_manager.cc:1918] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:27:33.979631 12424 log.cc:1196] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/58b1b58f1d57433c9175a881614cea4b
I20260809 09:27:33.979872 12424 ts_tablet_manager.cc:1939] T 58b1b58f1d57433c9175a881614cea4b P f6d2777a2a3a478ca3ba3faa864e8050: Deleting consensus metadata
I20260809 09:27:33.980070 11517 catalog_manager.cc:4841] TS 6e2d5f66c49d49c585a2a28c58235d3b (127.9.102.3:33743): tablet 58b1b58f1d57433c9175a881614cea4b (table test-table [id=56593fd2e1574b7ab292270b6e619b1b]) successfully deleted
I20260809 09:27:33.980412 11513 catalog_manager.cc:4841] TS f6d2777a2a3a478ca3ba3faa864e8050 (127.9.102.2:33999): tablet 58b1b58f1d57433c9175a881614cea4b (table test-table [id=56593fd2e1574b7ab292270b6e619b1b]) successfully deleted
I20260809 09:27:33.980535 12425 ts_tablet_manager.cc:1905] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:33.982654 12425 ts_tablet_manager.cc:1918] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:27:33.982715 12425 log.cc:1196] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786267646682796-9624-0/minicluster-data/ts-0/wal/wals/58b1b58f1d57433c9175a881614cea4b
I20260809 09:27:33.982967 12425 ts_tablet_manager.cc:1939] T 58b1b58f1d57433c9175a881614cea4b P ef82bc3c24db4178a08f8023d686208f: Deleting consensus metadata
Aug 09 09:27:33 dist-test-slave-7vjv krb5kdc[11463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267653, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.102.1@KRBTEST.COM
I20260809 09:27:33.983536 11513 catalog_manager.cc:4841] TS ef82bc3c24db4178a08f8023d686208f (127.9.102.1:37613): tablet 58b1b58f1d57433c9175a881614cea4b (table test-table [id=56593fd2e1574b7ab292270b6e619b1b]) successfully deleted
I20260809 09:27:33.987267 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.976810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.9.102.1:37613)
0809 09:27:33.976929 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.976933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.977385 (+ 452us) server_negotiation.cc:407] Connection header received
0809 09:27:33.977483 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.977486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.977527 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.977599 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:33.978412 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.979093 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.980594 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.980864 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.980975 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.983955 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:33.983962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:33.983975 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:33.983994 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:33.985625 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.986034 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.986038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.986039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.986111 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.986367 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.986369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.986370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.986600 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:33.986771 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.986942 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.987072 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260809 09:27:33.987646 11863 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39010
I20260809 09:27:33.996881 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:33.989149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.9.102.62:39319)
0809 09:27:33.989263 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:33.989283 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:33.989434 (+ 151us) server_negotiation.cc:407] Connection header received
0809 09:27:33.989570 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:33.989572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:33.989609 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:33.989690 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:33.990241 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.990693 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.991411 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:33.991720 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:33.991834 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:33.993599 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:33.993604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:33.993607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:33.993634 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:33.995283 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.995720 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.995724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.995726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.995816 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:33.996070 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:33.996073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:33.996075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:33.996282 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:33.996411 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:33.996632 (+ 221us) server_negotiation.cc:299] Negotiation successful
0809 09:27:33.996739 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260809 09:27:33.997208 11534 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38826
I20260809 09:27:33.998263 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 11662
I20260809 09:27:34.012248 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 11882
I20260809 09:27:34.044734 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 12114
I20260809 09:27:34.053184 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 11485
2026-08-09T09:27:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2820 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:27:34 dist-test-slave-7vjv krb5kdc[12486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:27:34 dist-test-slave-7vjv krb5kdc[12486](info): set up 2 sockets
Aug 09 09:27:34 dist-test-slave-7vjv krb5kdc[12486](info): commencing operation
krb5kdc: starting...
W20260809 09:27:35.207610 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:27:35 dist-test-slave-7vjv krb5kdc[12486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:27:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:27:35.317059 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:42791
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:42791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:35.415951 12513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:35.416165 12513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:35.416217 12513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:35.419579 12513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:35.419633 12513 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 09:27:35.419648 12513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:35.419662 12513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:35.419677 12513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:35.419724 12513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:35.424261 12513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:42791
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:42791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:35.425093 12513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:35.425971 12513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:35.433354 12522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:35.433320 12518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:35.433314 12520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:35.433410 12513 server_base.cc:1034] running on GCE node
I20260809 09:27:35.434265 12513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:35.435177 12513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:35.436329 12513 hybrid_clock.cc:648] HybridClock initialized: now 1786267655436306 us; error 36 us; skew 500 ppm
Aug 09 09:27:35 dist-test-slave-7vjv krb5kdc[12486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267655, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:35.438831 12513 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:27:35.439644 12513 webserver.cc:460] Webserver started at http://127.9.102.62:33041/ using document root <none> and password file <none>
I20260809 09:27:35.440186 12513 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:35.440275 12513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:35.440491 12513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:35.442301 12513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "213728ff271b45cbbf2e1696e914f6b3"
format_stamp: "Formatted at 2026-08-09 09:27:35 on dist-test-slave-7vjv"
server_key: "102157dfcaa0940833b2a84dfe579c79"
server_key_iv: "f716f329a7a2ee5bfbcd12c9fce16d8d"
server_key_version: "encryptionkey@0"
I20260809 09:27:35.442862 12513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "213728ff271b45cbbf2e1696e914f6b3"
format_stamp: "Formatted at 2026-08-09 09:27:35 on dist-test-slave-7vjv"
server_key: "102157dfcaa0940833b2a84dfe579c79"
server_key_iv: "f716f329a7a2ee5bfbcd12c9fce16d8d"
server_key_version: "encryptionkey@0"
I20260809 09:27:35.446105 12513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:35.448081 12531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:35.449121 12513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:35.449251 12513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "213728ff271b45cbbf2e1696e914f6b3"
format_stamp: "Formatted at 2026-08-09 09:27:35 on dist-test-slave-7vjv"
server_key: "102157dfcaa0940833b2a84dfe579c79"
server_key_iv: "f716f329a7a2ee5bfbcd12c9fce16d8d"
server_key_version: "encryptionkey@0"
I20260809 09:27:35.449350 12513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:35.466589 12513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:35.468886 12513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:35.469077 12513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:35.476092 12513 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:42791
I20260809 09:27:35.476145 12644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:42791 every 8 connection(s)
I20260809 09:27:35.476867 12513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:35.479167 12645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:35.482451 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 12513
I20260809 09:27:35.482570 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:35.482816 9624 external_mini_cluster.cc:1428] Setting key 3a0b7df5e08abe2219988267d47db653
I20260809 09:27:35.484668 12645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3: Bootstrap starting.
I20260809 09:27:35.487197 12645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:35.488047 12645 log.cc:824] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:35.490087 12645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3: No bootstrap required, opened a new log
Aug 09 09:27:35 dist-test-slave-7vjv krb5kdc[12486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:35.493723 12645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213728ff271b45cbbf2e1696e914f6b3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42791 } }
I20260809 09:27:35.493906 12645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:35.493960 12645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213728ff271b45cbbf2e1696e914f6b3, State: Initialized, Role: FOLLOWER
I20260809 09:27:35.494320 12645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213728ff271b45cbbf2e1696e914f6b3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42791 } }
I20260809 09:27:35.494428 12645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:35.494505 12645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:35.494624 12645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:35.495349 12645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213728ff271b45cbbf2e1696e914f6b3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42791 } }
I20260809 09:27:35.495661 12645 leader_election.cc:302] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 213728ff271b45cbbf2e1696e914f6b3; no voters:
I20260809 09:27:35.495997 12645 leader_election.cc:288] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:35.496174 12654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:35.496390 12654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [term 1 LEADER]: Becoming Leader. State: Replica: 213728ff271b45cbbf2e1696e914f6b3, State: Running, Role: LEADER
I20260809 09:27:35.496711 12654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213728ff271b45cbbf2e1696e914f6b3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42791 } }
I20260809 09:27:35.497267 12645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:35.497319 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:35.484035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.9.102.62:42791)
0809 09:27:35.484450 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:35.484467 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:35.484504 (+ 37us) server_negotiation.cc:407] Connection header received
0809 09:27:35.485279 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:35.485300 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:35.485679 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:35.486057 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:35.486712 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.488196 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.489208 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.489871 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.490032 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:35.492665 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:35.492669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:35.492680 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:35.492703 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:35.494382 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:35.494906 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:35.494911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:35.494916 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:35.495024 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:35.495286 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:35.495289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:35.495291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:35.495692 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:35.495942 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:35.496254 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 09:27:35.496561 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":107,"threads_started":1}
I20260809 09:27:35.498284 12657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "213728ff271b45cbbf2e1696e914f6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213728ff271b45cbbf2e1696e914f6b3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42791 } } }
I20260809 09:27:35.498411 12657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:35.498711 12657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 213728ff271b45cbbf2e1696e914f6b3. Latest consensus state: current_term: 1 leader_uuid: "213728ff271b45cbbf2e1696e914f6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213728ff271b45cbbf2e1696e914f6b3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42791 } } }
I20260809 09:27:35.498838 12657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3 [sys.catalog]: This master's current role is: LEADER
W20260809 09:27:35.501824 12679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:27:35.501907 12679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:27:35.501979 12668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:35.502657 12668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:35.507400 12668 catalog_manager.cc:1347] Generated new cluster ID: 122d369e83714d84968ec0d73938d6ed
I20260809 09:27:35.507452 12668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:35.545225 12668 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:35.545939 12668 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:35.551159 12668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3: Generated new TSK 0
I20260809 09:27:35.551720 12668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:27:35.582800 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:42791
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:35.683334 12686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:35.683534 12686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:35.683596 12686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:35.687186 12686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:35.687227 12686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:35.687261 12686 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 09:27:35.687274 12686 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 09:27:35.687301 12686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:27:35.691836 12686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.102.62:42791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:35.692644 12686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:35.693476 12686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:35.701952 12699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:35.702023 12686 server_base.cc:1034] running on GCE node
W20260809 09:27:35.701979 12696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:35.701980 12695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:35.702544 12686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:35.703083 12686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:35.704262 12686 hybrid_clock.cc:648] HybridClock initialized: now 1786267655704246 us; error 33 us; skew 500 ppm
Aug 09 09:27:35 dist-test-slave-7vjv krb5kdc[12486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267655, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:35.706746 12686 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:27:35.707480 12686 webserver.cc:460] Webserver started at http://127.9.102.1:44369/ using document root <none> and password file <none>
I20260809 09:27:35.707978 12686 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:35.708073 12686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:35.708312 12686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:35.710156 12686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "e4ae233a060640b7a4875d0c90451d26"
format_stamp: "Formatted at 2026-08-09 09:27:35 on dist-test-slave-7vjv"
server_key: "29620a11c1c9d0ad1b52bdc3d9ea9e90"
server_key_iv: "8b2898901e4ad259dae63ee671157b74"
server_key_version: "encryptionkey@0"
I20260809 09:27:35.710726 12686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "e4ae233a060640b7a4875d0c90451d26"
format_stamp: "Formatted at 2026-08-09 09:27:35 on dist-test-slave-7vjv"
server_key: "29620a11c1c9d0ad1b52bdc3d9ea9e90"
server_key_iv: "8b2898901e4ad259dae63ee671157b74"
server_key_version: "encryptionkey@0"
I20260809 09:27:35.714073 12686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:35.716199 12712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:35.717195 12686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:35.717326 12686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "e4ae233a060640b7a4875d0c90451d26"
format_stamp: "Formatted at 2026-08-09 09:27:35 on dist-test-slave-7vjv"
server_key: "29620a11c1c9d0ad1b52bdc3d9ea9e90"
server_key_iv: "8b2898901e4ad259dae63ee671157b74"
server_key_version: "encryptionkey@0"
I20260809 09:27:35.717427 12686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:35.732251 12686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:35.734671 12686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:35.734851 12686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:35.736068 12686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:35.736217 12686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:35.736280 12686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:35.736343 12686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 09:27:35 dist-test-slave-7vjv krb5kdc[12486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267655, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:35.746850 12686 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:35483
I20260809 09:27:35.746898 12910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:35483 every 8 connection(s)
I20260809 09:27:35.747826 12686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:27:35.748039 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 12686
I20260809 09:27:35.748198 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:27:35.748499 9624 external_mini_cluster.cc:1428] Setting key 0348203bebe3fa87317897e9f3c0b4ba
I20260809 09:27:35.750000 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:35.736785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:49003 (local address 127.9.102.62:42791)
0809 09:27:35.736934 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:35.736939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:35.737998 (+ 1059us) server_negotiation.cc:407] Connection header received
0809 09:27:35.739196 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:35.739199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:35.739255 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:35.739359 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:35.740556 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.741084 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.742137 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.742425 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.742534 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:35.745730 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:35.745734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:35.745737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:35.745764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:35.747365 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:35.747851 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:35.747856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:35.747861 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:35.747953 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:35.748412 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:35.748415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:35.748417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:35.748646 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:35.748749 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:35.749602 (+ 853us) server_negotiation.cc:299] Negotiation successful
0809 09:27:35.749728 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 09:27:35.750722 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:35.737451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:42791 (local address 127.9.102.1:49003)
0809 09:27:35.737867 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:35.737912 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:35.738960 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:35.739517 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:35.739524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:35.739833 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:35.740401 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:35.740416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.741208 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.741211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:35.742011 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:35.742021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.742587 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.742592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:35.742749 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:35.743308 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:35.743342 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:35.745585 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:35.747474 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:35.747478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:35.747487 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:35.747713 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:35.748069 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:35.748072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:35.748074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:35.748286 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:35.748904 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:35.748910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:35.749281 (+ 371us) client_negotiation.cc:769] Sending connection context
0809 09:27:35.749531 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 09:27:35.749843 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260809 09:27:35.752257 12912 heartbeater.cc:347] Connected to a master server at 127.9.102.62:42791
I20260809 09:27:35.752406 12912 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:35.752985 12912 heartbeater.cc:511] Master 127.9.102.62:42791 requested a full tablet report, sending...
I20260809 09:27:35.754670 12554 ts_manager.cc:194] Registered new tserver with Master: e4ae233a060640b7a4875d0c90451d26 (127.9.102.1:35483)
I20260809 09:27:35.755793 12554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:49003
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:27:35.763217 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:35.754483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:42791 (local address 127.9.102.1:53777)
0809 09:27:35.754796 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:35.754813 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:35.754910 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:35.756589 (+ 1679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:35.756592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:35.756811 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:35.756927 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:35.756935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.758060 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.758063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:35.758615 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:35.758623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.759223 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.759226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:35.759292 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:35.759783 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:35.759795 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:35.760077 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:35.761659 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:35.761661 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:35.761663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:35.761878 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:35.762175 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:35.762178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:35.762179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:35.762260 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:35.762766 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:35.762768 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:35.762846 (+ 78us) client_negotiation.cc:769] Sending connection context
0809 09:27:35.762938 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 09:27:35.763039 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
I20260809 09:27:35.763289 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:35.756230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:53777 (local address 127.9.102.62:42791)
0809 09:27:35.756355 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:35.756359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:35.756375 (+ 16us) server_negotiation.cc:407] Connection header received
0809 09:27:35.756411 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:35.756416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:35.756464 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:35.756551 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:35.757065 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.757922 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.758739 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.759099 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.759209 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:35.760253 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:35.760259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:35.760264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:35.760295 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:35.761577 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:35.761997 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:35.762000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:35.762001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:35.762081 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:35.762351 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:35.762353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:35.762354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:35.762659 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:35.762779 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:35.762997 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 09:27:35.763118 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:27:35.776578 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:42791
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:35.877353 12927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:35.877568 12927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:35.877609 12927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:35.881129 12927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:35.881171 12927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:35.881202 12927 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 09:27:35.881214 12927 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 09:27:35.881244 12927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:27:35.885754 12927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.102.62:42791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:35.886439 12927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:35.887157 12927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:35.895465 12942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:35.895610 12927 server_base.cc:1034] running on GCE node
W20260809 09:27:35.895491 12945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:35.895511 12943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:35.896507 12927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:35.897030 12927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:35.898211 12927 hybrid_clock.cc:648] HybridClock initialized: now 1786267655898177 us; error 48 us; skew 500 ppm
Aug 09 09:27:35 dist-test-slave-7vjv krb5kdc[12486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267655, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:35.900715 12927 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:27:35.901464 12927 webserver.cc:460] Webserver started at http://127.9.102.2:37859/ using document root <none> and password file <none>
I20260809 09:27:35.901938 12927 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:35.902047 12927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:35.902257 12927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:35.904017 12927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "5440fdfa70904747a6f7886cc25be34f"
format_stamp: "Formatted at 2026-08-09 09:27:35 on dist-test-slave-7vjv"
server_key: "6f1b83df493a8453d46a6f046ce8f8dc"
server_key_iv: "63e177068cd124c558472ff702e7a175"
server_key_version: "encryptionkey@0"
I20260809 09:27:35.904627 12927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "5440fdfa70904747a6f7886cc25be34f"
format_stamp: "Formatted at 2026-08-09 09:27:35 on dist-test-slave-7vjv"
server_key: "6f1b83df493a8453d46a6f046ce8f8dc"
server_key_iv: "63e177068cd124c558472ff702e7a175"
server_key_version: "encryptionkey@0"
I20260809 09:27:35.907874 12927 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:27:35.909999 12958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:35.910951 12927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 09:27:35.911077 12927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "5440fdfa70904747a6f7886cc25be34f"
format_stamp: "Formatted at 2026-08-09 09:27:35 on dist-test-slave-7vjv"
server_key: "6f1b83df493a8453d46a6f046ce8f8dc"
server_key_iv: "63e177068cd124c558472ff702e7a175"
server_key_version: "encryptionkey@0"
I20260809 09:27:35.911170 12927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:35.931960 12927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:35.934298 12927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:35.934490 12927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:35.935828 12927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:35.935904 12927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:35.935954 12927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:35.935985 12927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 09:27:35 dist-test-slave-7vjv krb5kdc[12486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267655, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:35.947345 12927 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:33841
I20260809 09:27:35.948241 13161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:33841 every 8 connection(s)
I20260809 09:27:35.948283 12927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:27:35.949587 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:35.936439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:60637 (local address 127.9.102.62:42791)
0809 09:27:35.936575 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:35.936579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:35.937962 (+ 1383us) server_negotiation.cc:407] Connection header received
0809 09:27:35.939062 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:35.939065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:35.939136 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:35.939221 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:35.940769 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.941503 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.942674 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.942972 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.943257 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:35.945858 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:35.945861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:35.945863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:35.945884 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:35.947541 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:35.948137 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:35.948140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:35.948142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:35.948225 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:35.948569 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:35.948573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:35.948574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:35.948800 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:35.948901 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:35.949371 (+ 470us) server_negotiation.cc:299] Negotiation successful
0809 09:27:35.949462 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:27:35.950230 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:35.937361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:42791 (local address 127.9.102.2:60637)
0809 09:27:35.937816 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:35.937852 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:35.938848 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:35.939424 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:35.939434 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:35.939888 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:35.940592 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:35.940616 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.941623 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.941625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:35.942553 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:35.942562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.943347 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.943353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:35.943501 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:35.944068 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:35.944087 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:35.945690 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:35.947658 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:35.947665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:35.947681 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:35.947975 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:35.948316 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:35.948321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:35.948322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:35.948474 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:35.948954 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:35.948961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:35.949164 (+ 203us) client_negotiation.cc:769] Sending connection context
0809 09:27:35.949303 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 09:27:35.949524 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":123,"threads_started":1}
I20260809 09:27:35.951328 13163 heartbeater.cc:347] Connected to a master server at 127.9.102.62:42791
I20260809 09:27:35.951471 13163 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:35.951692 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 12927
I20260809 09:27:35.951807 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:27:35.951911 13163 heartbeater.cc:511] Master 127.9.102.62:42791 requested a full tablet report, sending...
I20260809 09:27:35.952185 9624 external_mini_cluster.cc:1428] Setting key 4531a9f56310ae79fe40452e46c2d2f6
I20260809 09:27:35.952970 12555 ts_manager.cc:194] Registered new tserver with Master: 5440fdfa70904747a6f7886cc25be34f (127.9.102.2:33841)
I20260809 09:27:35.953647 12555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:60637
I20260809 09:27:35.960500 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:35.953532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:40761 (local address 127.9.102.62:42791)
0809 09:27:35.953640 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:35.953643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:35.953859 (+ 216us) server_negotiation.cc:407] Connection header received
0809 09:27:35.953993 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:35.953995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:35.954030 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:35.954112 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:35.954535 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.955234 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.956141 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.956325 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.956401 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:35.957460 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:35.957463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:35.957464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:35.957484 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:35.958875 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:35.959306 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:35.959308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:35.959309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:35.959373 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:35.959706 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:35.959708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:35.959709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:35.959888 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:35.959963 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:35.960301 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 09:27:35.960386 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 09:27:35.960647 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:35.953506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:42791 (local address 127.9.102.2:40761)
0809 09:27:35.953769 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:35.953782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:35.953883 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:35.954127 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:35.954129 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:35.954310 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:35.954412 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:35.954419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.955367 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.955369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:35.955998 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:35.956009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:35.956425 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:35.956429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:35.956524 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:35.956928 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:35.956945 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:35.957328 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:35.958984 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:35.958987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:35.958989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:35.959219 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:35.959473 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:35.959477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:35.959480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:35.959611 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:35.959988 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:35.959994 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:35.960129 (+ 135us) client_negotiation.cc:769] Sending connection context
0809 09:27:35.960255 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 09:27:35.960409 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:27:35.978904 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:42791
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:36.081986 13179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:36.082218 13179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:36.082293 13179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:36.085700 13179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:36.085768 13179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:36.085829 13179 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 09:27:36.085868 13179 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 09:27:36.085934 13179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:27:36.090363 13179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.102.62:42791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:36.091163 13179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:36.092025 13179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:36.100600 13187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:36.100662 13191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:36.100600 13186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:36.100703 13179 server_base.cc:1034] running on GCE node
I20260809 09:27:36.101176 13179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:36.101740 13179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:36.102886 13179 hybrid_clock.cc:648] HybridClock initialized: now 1786267656102869 us; error 27 us; skew 500 ppm
Aug 09 09:27:36 dist-test-slave-7vjv krb5kdc[12486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267656, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:36.105448 13179 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:27:36.106206 13179 webserver.cc:460] Webserver started at http://127.9.102.3:41121/ using document root <none> and password file <none>
I20260809 09:27:36.106686 13179 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:36.106765 13179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:36.106980 13179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:36.108743 13179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "1cfa683de1cd49c6ae7c48cab6d1f732"
format_stamp: "Formatted at 2026-08-09 09:27:36 on dist-test-slave-7vjv"
server_key: "b20a0b3d47640cc17e83199d7743d5fb"
server_key_iv: "3aa410255a5bae1052a85ea06f62333d"
server_key_version: "encryptionkey@0"
I20260809 09:27:36.109220 13179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "1cfa683de1cd49c6ae7c48cab6d1f732"
format_stamp: "Formatted at 2026-08-09 09:27:36 on dist-test-slave-7vjv"
server_key: "b20a0b3d47640cc17e83199d7743d5fb"
server_key_iv: "3aa410255a5bae1052a85ea06f62333d"
server_key_version: "encryptionkey@0"
I20260809 09:27:36.112702 13179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:27:36.114794 13206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:36.115845 13179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:36.115965 13179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "1cfa683de1cd49c6ae7c48cab6d1f732"
format_stamp: "Formatted at 2026-08-09 09:27:36 on dist-test-slave-7vjv"
server_key: "b20a0b3d47640cc17e83199d7743d5fb"
server_key_iv: "3aa410255a5bae1052a85ea06f62333d"
server_key_version: "encryptionkey@0"
I20260809 09:27:36.116067 13179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:36.137038 13179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:36.139384 13179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:36.139566 13179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:36.140899 13179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:36.140975 13179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:36.141044 13179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:36.141094 13179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 09:27:36 dist-test-slave-7vjv krb5kdc[12486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267656, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:36.150920 13179 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:34289
I20260809 09:27:36.150991 13400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:34289 every 8 connection(s)
I20260809 09:27:36.151914 13179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:27:36.153671 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.141432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:57909 (local address 127.9.102.62:42791)
0809 09:27:36.141597 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.141601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.142726 (+ 1125us) server_negotiation.cc:407] Connection header received
0809 09:27:36.143650 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.143653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.143712 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.143827 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:36.145047 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.145568 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.146685 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.146905 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.147019 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.149741 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:36.149746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:36.149749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:36.149783 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:36.151460 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:36.152092 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:36.152096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:36.152097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:36.152202 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:36.152643 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:36.152648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:36.152649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:36.152866 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:36.152962 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.153462 (+ 500us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.153567 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 09:27:36.154284 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 13179
I20260809 09:27:36.154305 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.142195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:42791 (local address 127.9.102.3:57909)
0809 09:27:36.142584 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:36.142626 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:36.143445 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:36.144053 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:36.144061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:36.144386 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:36.144913 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.144925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.145686 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.145690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.146555 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.146566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.147072 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.147076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.147273 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.147861 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:36.147886 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:36.149564 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:36.151597 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:36.151602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:36.151614 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:36.151962 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:36.152314 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:36.152317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:36.152319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:36.152494 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:36.152972 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:36.152978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:36.153224 (+ 246us) client_negotiation.cc:769] Sending connection context
0809 09:27:36.153408 (+ 184us) client_negotiation.cc:240] Negotiation successful
0809 09:27:36.153621 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":118,"threads_started":1}
I20260809 09:27:36.154399 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:27:36.154636 9624 external_mini_cluster.cc:1428] Setting key 982021176d4e26eb54a933b75d69ffd1
I20260809 09:27:36.155566 13402 heartbeater.cc:347] Connected to a master server at 127.9.102.62:42791
I20260809 09:27:36.155726 13402 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:36.156225 13402 heartbeater.cc:511] Master 127.9.102.62:42791 requested a full tablet report, sending...
I20260809 09:27:36.157147 12554 ts_manager.cc:194] Registered new tserver with Master: 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289)
I20260809 09:27:36.157873 12554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:57909
I20260809 09:27:36.162783 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.157345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:50601 (local address 127.9.102.62:42791)
0809 09:27:36.157465 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.157469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.157636 (+ 167us) server_negotiation.cc:407] Connection header received
0809 09:27:36.157723 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.157725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.157766 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.157820 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:36.158307 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.158907 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.159581 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.159761 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.159830 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.160693 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:36.160696 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:36.160697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:36.160717 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:36.161692 (+ 975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:36.162012 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:36.162014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:36.162015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:36.162082 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:36.162283 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:36.162285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:36.162286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:36.162409 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:36.162472 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.162633 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.162704 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 09:27:36.162796 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.157312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:42791 (local address 127.9.102.3:50601)
0809 09:27:36.157554 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:36.157567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:36.157655 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:36.157876 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:36.157878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:36.158097 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:36.158220 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.158226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.159017 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.159019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.159479 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.159486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.159846 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.159849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.159922 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.160307 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:36.160318 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:36.160583 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:36.161774 (+ 1191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:36.161776 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:36.161777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:36.161939 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:36.162150 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:36.162152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:36.162153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:36.162217 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:36.162476 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:36.162478 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:36.162538 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 09:27:36.162603 (+ 65us) client_negotiation.cc:240] Negotiation successful
0809 09:27:36.162684 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
I20260809 09:27:36.168311 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 09:27:36 dist-test-slave-7vjv krb5kdc[12486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 09:27:36 dist-test-slave-7vjv krb5kdc[12486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:36.191174 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.184299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.9.102.62:42791)
0809 09:27:36.184420 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.184422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.184431 (+ 9us) server_negotiation.cc:407] Connection header received
0809 09:27:36.184455 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.184457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.184483 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.184555 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:36.185151 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.185583 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.186328 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.186514 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.186610 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.188735 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:36.188737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:36.188739 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:36.188757 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:36.189975 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:36.190324 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:36.190326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:36.190327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:36.190390 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:36.190598 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:36.190600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:36.190601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:36.190741 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:36.190824 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.190991 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.191071 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 09:27:36.193463 12554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:27:36.196079 12554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:27:36.208599 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.203309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:34289 (local address 127.0.0.1:56444)
0809 09:27:36.203812 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:36.203825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:36.203910 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:36.204986 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:36.204989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:36.205025 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:36.205103 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.205110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.206335 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.206338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.207331 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.207340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.208084 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.208087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.208212 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.208251 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 09:27:36.208373 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 09:27:36.208473 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260809 09:27:36.208595 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.204155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.9.102.3:34289)
0809 09:27:36.204487 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.204497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.204512 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:27:36.204572 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.204576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.204705 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.204846 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:36.205228 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.206216 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.207441 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.207956 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.208080 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.208236 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.208404 (+ 168us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.208481 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":99,"threads_started":1}
I20260809 09:27:36.208822 13429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.203084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:33841 (local address 127.0.0.1:43222)
0809 09:27:36.203393 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:36.203437 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:36.203552 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:36.204435 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:36.204440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:36.204459 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:36.204588 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.204596 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.206102 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.206105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.207519 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.207524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.208433 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.208436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.208565 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.208584 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 09:27:36.208640 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 09:27:36.208705 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":235,"thread_start_us":66,"threads_started":1}
I20260809 09:27:36.209307 13430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.203446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:35483 (local address 127.0.0.1:58902)
0809 09:27:36.204081 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:36.204111 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:36.204191 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:36.204839 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:36.204841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:36.204856 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:36.204938 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.204943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.205881 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.205885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.207415 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.207427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.208433 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.208436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.209007 (+ 571us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.209034 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 09:27:36.209121 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 09:27:36.209183 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":600,"thread_start_us":88,"threads_started":1}
I20260809 09:27:36.209403 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.203766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58902 (local address 127.9.102.1:35483)
0809 09:27:36.204438 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.204448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.204468 (+ 20us) server_negotiation.cc:407] Connection header received
0809 09:27:36.204541 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.204546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.204705 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.204846 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:36.205062 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.205767 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.207516 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.208303 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.208553 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.208705 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.209163 (+ 458us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.209266 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":109,"threads_started":1}
I20260809 09:27:36.209935 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.203650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43222 (local address 127.9.102.2:33841)
0809 09:27:36.203947 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.203958 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.203980 (+ 22us) server_negotiation.cc:407] Connection header received
0809 09:27:36.204056 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.204061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.204275 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.204742 (+ 467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:36.204893 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.205956 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.207616 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.208303 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.209554 (+ 1251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.209694 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.209750 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.209811 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":127,"threads_started":1}
I20260809 09:27:36.211503 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 4f926b8536e7461cb8799290a60617bb (DEFAULT_TABLE table=test-table [id=4dd568398792464ca9a8c09d1e9bb6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:36.211894 13041 tablet_service.cc:1467] Processing CreateTablet for tablet 4f926b8536e7461cb8799290a60617bb (DEFAULT_TABLE table=test-table [id=4dd568398792464ca9a8c09d1e9bb6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:36.212119 12802 tablet_service.cc:1467] Processing CreateTablet for tablet 4f926b8536e7461cb8799290a60617bb (DEFAULT_TABLE table=test-table [id=4dd568398792464ca9a8c09d1e9bb6c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:36.212641 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f926b8536e7461cb8799290a60617bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:36.212744 13041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f926b8536e7461cb8799290a60617bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:36.212932 12802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f926b8536e7461cb8799290a60617bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:36.219380 13438 tablet_bootstrap.cc:492] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f: Bootstrap starting.
I20260809 09:27:36.220068 13439 tablet_bootstrap.cc:492] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732: Bootstrap starting.
I20260809 09:27:36.220156 13440 tablet_bootstrap.cc:492] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26: Bootstrap starting.
I20260809 09:27:36.221534 13438 tablet_bootstrap.cc:654] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:36.222023 13439 tablet_bootstrap.cc:654] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:36.222054 13440 tablet_bootstrap.cc:654] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:36.222294 13438 log.cc:824] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:36.222769 13440 log.cc:824] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:36.222769 13439 log.cc:824] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:36.224149 13438 tablet_bootstrap.cc:492] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f: No bootstrap required, opened a new log
I20260809 09:27:36.224346 13438 ts_tablet_manager.cc:1397] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:27:36.224385 13440 tablet_bootstrap.cc:492] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26: No bootstrap required, opened a new log
I20260809 09:27:36.224563 13440 ts_tablet_manager.cc:1397] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260809 09:27:36.224924 13439 tablet_bootstrap.cc:492] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732: No bootstrap required, opened a new log
I20260809 09:27:36.225078 13439 ts_tablet_manager.cc:1397] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260809 09:27:36.226903 13438 raft_consensus.cc:348] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } }
I20260809 09:27:36.227105 13438 raft_consensus.cc:374] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:36.227165 13438 raft_consensus.cc:729] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5440fdfa70904747a6f7886cc25be34f, State: Initialized, Role: FOLLOWER
I20260809 09:27:36.227571 13438 consensus_queue.cc:260] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } }
I20260809 09:27:36.227888 13440 raft_consensus.cc:348] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } }
I20260809 09:27:36.228116 13440 raft_consensus.cc:374] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:36.228163 13440 raft_consensus.cc:729] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ae233a060640b7a4875d0c90451d26, State: Initialized, Role: FOLLOWER
I20260809 09:27:36.228171 13163 heartbeater.cc:503] Master 127.9.102.62:42791 was elected leader, sending a full tablet report...
I20260809 09:27:36.228467 13438 ts_tablet_manager.cc:1428] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260809 09:27:36.228345 13439 raft_consensus.cc:348] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } }
I20260809 09:27:36.228526 13439 raft_consensus.cc:374] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:36.228571 13439 raft_consensus.cc:729] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfa683de1cd49c6ae7c48cab6d1f732, State: Initialized, Role: FOLLOWER
I20260809 09:27:36.228526 13440 consensus_queue.cc:260] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } }
I20260809 09:27:36.228921 13439 consensus_queue.cc:260] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } }
I20260809 09:27:36.229071 12912 heartbeater.cc:503] Master 127.9.102.62:42791 was elected leader, sending a full tablet report...
I20260809 09:27:36.229429 13440 ts_tablet_manager.cc:1428] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260809 09:27:36.229861 13439 ts_tablet_manager.cc:1428] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:27:36.229920 13402 heartbeater.cc:503] Master 127.9.102.62:42791 was elected leader, sending a full tablet report...
W20260809 09:27:36.249567 12914 tablet.cc:2369] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:27:36.402892 13403 tablet.cc:2369] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:27:36.436241 13448 raft_consensus.cc:482] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:27:36.436444 13448 raft_consensus.cc:504] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } }
I20260809 09:27:36.437587 13448 leader_election.cc:288] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5440fdfa70904747a6f7886cc25be34f (127.9.102.2:33841), 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289)
I20260809 09:27:36.441605 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.438043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:40783 (local address 127.9.102.3:34289)
0809 09:27:36.438186 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.438191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.438257 (+ 66us) server_negotiation.cc:407] Connection header received
0809 09:27:36.438394 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.438397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.438438 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.438559 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:36.438852 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.439321 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.440586 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.440973 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.441059 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.441132 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.441388 (+ 256us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.441465 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 09:27:36.441617 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.437836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:34723 (local address 127.9.102.2:33841)
0809 09:27:36.438006 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.438011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.438133 (+ 122us) server_negotiation.cc:407] Connection header received
0809 09:27:36.438350 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.438355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.438412 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.438487 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:36.438804 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.439314 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.440578 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.441049 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.441155 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.441223 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.441435 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.441511 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 09:27:36.441617 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.437822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:34289 (local address 127.9.102.1:40783)
0809 09:27:36.438177 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:36.438188 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:36.438294 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:36.438586 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:36.438592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:36.438610 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:36.438725 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.438729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.439453 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.439456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.440450 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.440460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.441094 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.441099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.441230 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.441265 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 09:27:36.441338 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 09:27:36.441414 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":119,"thread_start_us":85,"threads_started":1}
I20260809 09:27:36.441629 13450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.437728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:33841 (local address 127.9.102.1:34723)
0809 09:27:36.438007 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:36.438022 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:36.438217 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:36.438522 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:36.438527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:36.438547 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:36.438685 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.438691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.439423 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.439427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.440456 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.440465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.441177 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.441180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.441280 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.441310 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 09:27:36.441405 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 09:27:36.441465 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":98,"thread_start_us":88,"threads_started":1}
I20260809 09:27:36.442320 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f926b8536e7461cb8799290a60617bb" candidate_uuid: "e4ae233a060640b7a4875d0c90451d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" is_pre_election: true
I20260809 09:27:36.442363 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f926b8536e7461cb8799290a60617bb" candidate_uuid: "e4ae233a060640b7a4875d0c90451d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5440fdfa70904747a6f7886cc25be34f" is_pre_election: true
I20260809 09:27:36.442600 13071 raft_consensus.cc:2454] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ae233a060640b7a4875d0c90451d26 in term 0.
I20260809 09:27:36.442600 13323 raft_consensus.cc:2454] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ae233a060640b7a4875d0c90451d26 in term 0.
I20260809 09:27:36.443080 12717 leader_election.cc:302] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cfa683de1cd49c6ae7c48cab6d1f732, 5440fdfa70904747a6f7886cc25be34f, e4ae233a060640b7a4875d0c90451d26; no voters:
I20260809 09:27:36.443316 13448 raft_consensus.cc:2781] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:27:36.443394 13448 raft_consensus.cc:482] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:27:36.443423 13448 raft_consensus.cc:3037] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:36.444482 13448 raft_consensus.cc:504] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } }
I20260809 09:27:36.444810 13448 leader_election.cc:288] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [CANDIDATE]: Term 1 election: Requested vote from peers 5440fdfa70904747a6f7886cc25be34f (127.9.102.2:33841), 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289)
I20260809 09:27:36.445112 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f926b8536e7461cb8799290a60617bb" candidate_uuid: "e4ae233a060640b7a4875d0c90451d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5440fdfa70904747a6f7886cc25be34f"
I20260809 09:27:36.445147 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f926b8536e7461cb8799290a60617bb" candidate_uuid: "e4ae233a060640b7a4875d0c90451d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732"
I20260809 09:27:36.445233 13071 raft_consensus.cc:3037] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:36.445252 13323 raft_consensus.cc:3037] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:36.446249 13071 raft_consensus.cc:2454] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ae233a060640b7a4875d0c90451d26 in term 1.
I20260809 09:27:36.446566 12717 leader_election.cc:302] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5440fdfa70904747a6f7886cc25be34f, e4ae233a060640b7a4875d0c90451d26; no voters:
I20260809 09:27:36.446776 13448 raft_consensus.cc:2781] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:36.446792 13323 raft_consensus.cc:2454] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ae233a060640b7a4875d0c90451d26 in term 1.
I20260809 09:27:36.446961 13448 raft_consensus.cc:686] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 1 LEADER]: Becoming Leader. State: Replica: e4ae233a060640b7a4875d0c90451d26, State: Running, Role: LEADER
I20260809 09:27:36.447320 13448 consensus_queue.cc:237] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } }
W20260809 09:27:36.449309 13164 tablet.cc:2369] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:27:36.450435 12554 catalog_manager.cc:5495] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4ae233a060640b7a4875d0c90451d26 (127.9.102.1). New cstate: current_term: 1 leader_uuid: "e4ae233a060640b7a4875d0c90451d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } health_report { overall_health: HEALTHY } } }
Aug 09 09:27:36 dist-test-slave-7vjv krb5kdc[12486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267655, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:36.516434 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.509368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:42791 (local address 127.0.0.1:55034)
0809 09:27:36.509604 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:36.509615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:36.509696 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:36.509901 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:36.509903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:36.510230 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:36.510318 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.510323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.510950 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.510952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.511446 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.511453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.511781 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.511783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.511849 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.512405 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:36.512416 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:36.513667 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:36.515086 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:36.515094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:36.515097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:36.515325 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:36.515551 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:36.515553 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:36.515554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:36.515623 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:36.515940 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:36.515944 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:36.516031 (+ 87us) client_negotiation.cc:769] Sending connection context
0809 09:27:36.516177 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 09:27:36.516284 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":100,"threads_started":1}
I20260809 09:27:36.516446 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.509406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.9.102.62:42791)
0809 09:27:36.509516 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.509520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.509678 (+ 158us) server_negotiation.cc:407] Connection header received
0809 09:27:36.509760 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.509762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.509797 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.509858 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:36.510422 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.510865 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.511529 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.511697 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.511759 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.513774 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:36.513777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:36.513778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:36.513797 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:36.514966 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:36.515397 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:36.515399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:36.515400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:36.515480 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:36.515716 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:36.515718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:36.515719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:36.515859 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:36.515926 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.516221 (+ 295us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.516337 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":39}
I20260809 09:27:36.519634 12554 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:55034:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 09:27:36.520421 12554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:27:36.524785 12802 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1b04e2b20640138a66a2b579359c26 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4a4928d0d1548909fa83fb492ebbccd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 09:27:36.524863 13041 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1b04e2b20640138a66a2b579359c26 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4a4928d0d1548909fa83fb492ebbccd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 09:27:36.524924 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1b04e2b20640138a66a2b579359c26 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4a4928d0d1548909fa83fb492ebbccd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 09:27:36.525439 13041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1b04e2b20640138a66a2b579359c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:36.525439 12802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1b04e2b20640138a66a2b579359c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:36.525465 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1b04e2b20640138a66a2b579359c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:36.527624 13439 tablet_bootstrap.cc:492] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732: Bootstrap starting.
I20260809 09:27:36.528107 13440 tablet_bootstrap.cc:492] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: Bootstrap starting.
I20260809 09:27:36.528649 13439 tablet_bootstrap.cc:654] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:36.529510 13440 tablet_bootstrap.cc:654] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:36.530036 13439 tablet_bootstrap.cc:492] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732: No bootstrap required, opened a new log
I20260809 09:27:36.530131 13439 ts_tablet_manager.cc:1397] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 09:27:36.530154 13438 tablet_bootstrap.cc:492] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f: Bootstrap starting.
I20260809 09:27:36.530584 13439 raft_consensus.cc:348] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.530704 13439 raft_consensus.cc:374] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:36.530735 13439 raft_consensus.cc:729] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfa683de1cd49c6ae7c48cab6d1f732, State: Initialized, Role: FOLLOWER
I20260809 09:27:36.530823 13440 tablet_bootstrap.cc:492] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: No bootstrap required, opened a new log
I20260809 09:27:36.530848 13439 consensus_queue.cc:260] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.530898 13440 ts_tablet_manager.cc:1397] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 09:27:36.530999 13438 tablet_bootstrap.cc:654] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:36.531101 13439 ts_tablet_manager.cc:1428] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 09:27:36.531286 13440 raft_consensus.cc:348] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.531361 13440 raft_consensus.cc:374] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:36.531379 13440 raft_consensus.cc:729] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ae233a060640b7a4875d0c90451d26, State: Initialized, Role: FOLLOWER
I20260809 09:27:36.531625 13449 tablet_replica.cc:439] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } }
I20260809 09:27:36.531765 13449 tablet_replica.cc:442] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:36.532189 13440 consensus_queue.cc:260] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.532395 13438 tablet_bootstrap.cc:492] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f: No bootstrap required, opened a new log
I20260809 09:27:36.532469 13438 ts_tablet_manager.cc:1397] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:36.532433 13448 tablet_replica.cc:439] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } }
I20260809 09:27:36.532480 13440 ts_tablet_manager.cc:1428] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:36.532560 13448 tablet_replica.cc:442] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:36.532871 13438 raft_consensus.cc:348] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.532971 13438 raft_consensus.cc:374] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:36.533005 13438 raft_consensus.cc:729] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5440fdfa70904747a6f7886cc25be34f, State: Initialized, Role: FOLLOWER
I20260809 09:27:36.533119 13438 consensus_queue.cc:260] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.533357 13438 ts_tablet_manager.cc:1428] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 09:27:36.533643 13447 tablet_replica.cc:439] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } }
I20260809 09:27:36.533776 13447 tablet_replica.cc:442] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:36.567817 13448 raft_consensus.cc:482] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:27:36.567921 13448 raft_consensus.cc:504] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.568341 13448 leader_election.cc:288] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5440fdfa70904747a6f7886cc25be34f (127.9.102.2:33841), 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289)
I20260809 09:27:36.568673 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1b04e2b20640138a66a2b579359c26" candidate_uuid: "e4ae233a060640b7a4875d0c90451d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5440fdfa70904747a6f7886cc25be34f" is_pre_election: true
I20260809 09:27:36.568722 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1b04e2b20640138a66a2b579359c26" candidate_uuid: "e4ae233a060640b7a4875d0c90451d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" is_pre_election: true
I20260809 09:27:36.568815 13071 raft_consensus.cc:2454] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ae233a060640b7a4875d0c90451d26 in term 0.
I20260809 09:27:36.568851 13323 raft_consensus.cc:2454] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ae233a060640b7a4875d0c90451d26 in term 0.
I20260809 09:27:36.569083 12717 leader_election.cc:302] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5440fdfa70904747a6f7886cc25be34f, e4ae233a060640b7a4875d0c90451d26; no voters:
I20260809 09:27:36.569262 13448 raft_consensus.cc:2781] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:27:36.569324 13448 raft_consensus.cc:482] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:27:36.569350 13448 raft_consensus.cc:3037] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:36.570250 13448 raft_consensus.cc:504] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.570537 13448 leader_election.cc:288] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [CANDIDATE]: Term 1 election: Requested vote from peers 5440fdfa70904747a6f7886cc25be34f (127.9.102.2:33841), 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289)
I20260809 09:27:36.570854 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1b04e2b20640138a66a2b579359c26" candidate_uuid: "e4ae233a060640b7a4875d0c90451d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5440fdfa70904747a6f7886cc25be34f"
I20260809 09:27:36.570876 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1b04e2b20640138a66a2b579359c26" candidate_uuid: "e4ae233a060640b7a4875d0c90451d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732"
I20260809 09:27:36.570988 13071 raft_consensus.cc:3037] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:36.570998 13323 raft_consensus.cc:3037] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:36.572022 13323 raft_consensus.cc:2454] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ae233a060640b7a4875d0c90451d26 in term 1.
I20260809 09:27:36.572022 13071 raft_consensus.cc:2454] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ae233a060640b7a4875d0c90451d26 in term 1.
I20260809 09:27:36.572328 12715 leader_election.cc:302] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfa683de1cd49c6ae7c48cab6d1f732, e4ae233a060640b7a4875d0c90451d26; no voters:
I20260809 09:27:36.572499 13448 raft_consensus.cc:2781] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:36.572613 13448 raft_consensus.cc:686] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [term 1 LEADER]: Becoming Leader. State: Replica: e4ae233a060640b7a4875d0c90451d26, State: Running, Role: LEADER
I20260809 09:27:36.572728 13448 consensus_queue.cc:237] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.573365 13448 tablet_replica.cc:439] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: TxnStatusTablet state changed. Reason: New leader e4ae233a060640b7a4875d0c90451d26. Latest consensus state: current_term: 1 leader_uuid: "e4ae233a060640b7a4875d0c90451d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } }
I20260809 09:27:36.573458 13448 tablet_replica.cc:442] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: This TxnStatusTablet replica's current role is: LEADER
I20260809 09:27:36.573742 13478 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 09:27:36.573984 13478 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 09:27:36.574394 12552 catalog_manager.cc:5495] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4ae233a060640b7a4875d0c90451d26 (127.9.102.1). New cstate: current_term: 1 leader_uuid: "e4ae233a060640b7a4875d0c90451d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } health_report { overall_health: UNKNOWN } } }
I20260809 09:27:36.586719 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.582618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:35483 (local address 127.0.0.1:58914)
0809 09:27:36.582735 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:36.582750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:36.582843 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:36.583171 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:36.583173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:36.583180 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 09:27:36.583280 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.583284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.584498 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.584501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.585086 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.585094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.585690 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.585692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.585776 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.585816 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 09:27:36.586494 (+ 678us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 09:27:36.586504 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 09:27:36.586550 (+ 46us) client_negotiation.cc:240] Negotiation successful
0809 09:27:36.586614 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260809 09:27:36.586746 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.582690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.9.102.1:35483)
0809 09:27:36.582798 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.582802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.582818 (+ 16us) server_negotiation.cc:407] Connection header received
0809 09:27:36.582970 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.582974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.583056 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.583165 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:36.583484 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.584368 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.585201 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.585553 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.585678 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.585946 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:36.586401 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:36.586517 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.586564 (+ 47us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.586618 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 09:27:36.591157 13323 raft_consensus.cc:1261] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Refusing update from remote peer e4ae233a060640b7a4875d0c90451d26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:36.591181 13071 raft_consensus.cc:1261] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Refusing update from remote peer e4ae233a060640b7a4875d0c90451d26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:36.591818 13454 consensus_queue.cc:1035] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:36.592015 13448 consensus_queue.cc:1035] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:36.595592 13447 tablet_replica.cc:439] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f: TxnStatusTablet state changed. Reason: New leader e4ae233a060640b7a4875d0c90451d26. Latest consensus state: current_term: 1 leader_uuid: "e4ae233a060640b7a4875d0c90451d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } }
I20260809 09:27:36.595685 13447 tablet_replica.cc:442] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:36.595897 13449 tablet_replica.cc:439] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732: TxnStatusTablet state changed. Reason: New leader e4ae233a060640b7a4875d0c90451d26. Latest consensus state: current_term: 1 leader_uuid: "e4ae233a060640b7a4875d0c90451d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } }
I20260809 09:27:36.595999 13449 tablet_replica.cc:442] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:36.598559 13448 tablet_replica.cc:439] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4ae233a060640b7a4875d0c90451d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } }
I20260809 09:27:36.598668 13448 tablet_replica.cc:442] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: This TxnStatusTablet replica's current role is: LEADER
I20260809 09:27:36.598747 13484 mvcc.cc:205] Tried to move back new op lower bound from 7316552321391591424 to 7316552321323122688. Current Snapshot: MvccSnapshot[applied={T|T < 7316552321391591424}]
I20260809 09:27:36.599193 13449 tablet_replica.cc:439] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4ae233a060640b7a4875d0c90451d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } }
I20260809 09:27:36.599304 13449 tablet_replica.cc:442] T 4f1b04e2b20640138a66a2b579359c26 P 1cfa683de1cd49c6ae7c48cab6d1f732: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:36.599993 13491 mvcc.cc:205] Tried to move back new op lower bound from 7316552321391591424 to 7316552321323122688. Current Snapshot: MvccSnapshot[applied={T|T < 7316552321391591424}]
I20260809 09:27:36.600449 13448 tablet_replica.cc:439] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e4ae233a060640b7a4875d0c90451d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } }
I20260809 09:27:36.600540 13448 tablet_replica.cc:442] T 4f1b04e2b20640138a66a2b579359c26 P e4ae233a060640b7a4875d0c90451d26: This TxnStatusTablet replica's current role is: LEADER
I20260809 09:27:36.601845 13447 tablet_replica.cc:439] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e4ae233a060640b7a4875d0c90451d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } } }
I20260809 09:27:36.601933 13447 tablet_replica.cc:442] T 4f1b04e2b20640138a66a2b579359c26 P 5440fdfa70904747a6f7886cc25be34f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:27:36.610404 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.607322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.9.102.1:35483)
0809 09:27:36.607414 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.607418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.607484 (+ 66us) server_negotiation.cc:407] Connection header received
0809 09:27:36.607565 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.607568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.607607 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.607680 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:36.607890 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.608647 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.609420 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.609732 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.609820 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.610016 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:36.610099 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:36.610174 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.610255 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.610297 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 09:27:36.621037 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.617940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:44119 (local address 127.9.102.1:35483)
0809 09:27:36.618046 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.618050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.618214 (+ 164us) server_negotiation.cc:407] Connection header received
0809 09:27:36.618326 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.618332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.618374 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.618457 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:36.618663 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.619071 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.619989 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.620289 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.620406 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.620674 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:36.620749 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:36.620814 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.620894 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.620927 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260809 09:27:36.621085 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.617862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:35483 (local address 127.9.102.1:44119)
0809 09:27:36.618141 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:36.618151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:36.618255 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:36.618463 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:36.618466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:36.618471 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 09:27:36.618582 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.618588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.619167 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.619171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.619886 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.619896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.620406 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.620409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.620500 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.620533 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 09:27:36.620829 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 09:27:36.620835 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 09:27:36.620886 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 09:27:36.620949 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":18,"thread_start_us":81,"threads_started":1}
I20260809 09:27:36.626962 13323 raft_consensus.cc:1261] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Refusing update from remote peer e4ae233a060640b7a4875d0c90451d26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:36.627046 13071 raft_consensus.cc:1261] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Refusing update from remote peer e4ae233a060640b7a4875d0c90451d26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:36.627401 13512 consensus_queue.cc:1035] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:36.627568 13454 consensus_queue.cc:1035] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:36.666240 12554 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:36.666390 12554 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:36.668402 12554 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3: Sending DeleteTablet for 3 replicas of tablet 4f926b8536e7461cb8799290a60617bb
I20260809 09:27:36.668926 13041 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f926b8536e7461cb8799290a60617bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:36 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:43222
I20260809 09:27:36.669344 13525 tablet_replica.cc:330] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f: stopping tablet replica
I20260809 09:27:36.669317 13289 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f926b8536e7461cb8799290a60617bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:36 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:56444
I20260809 09:27:36.669618 13525 raft_consensus.cc:2229] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:36.669644 12802 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f926b8536e7461cb8799290a60617bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:36 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:58902
I20260809 09:27:36.669811 13525 raft_consensus.cc:2258] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:36.669771 12554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 09:27:36.669987 13526 tablet_replica.cc:330] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732: stopping tablet replica
W20260809 09:27:36.670117 12554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:27:36.670219 13526 raft_consensus.cc:2229] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:36.670369 13526 raft_consensus.cc:2258] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:36.670601 13527 tablet_replica.cc:330] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26: stopping tablet replica
I20260809 09:27:36.670810 13527 raft_consensus.cc:2229] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:27:36.670996 13525 ts_tablet_manager.cc:1905] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:36.671058 13527 raft_consensus.cc:2258] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:36.672184 13526 ts_tablet_manager.cc:1905] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:36.673874 13525 ts_tablet_manager.cc:1918] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:27:36.673964 13525 log.cc:1196] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/4f926b8536e7461cb8799290a60617bb
I20260809 09:27:36.674284 13525 ts_tablet_manager.cc:1939] T 4f926b8536e7461cb8799290a60617bb P 5440fdfa70904747a6f7886cc25be34f: Deleting consensus metadata
I20260809 09:27:36.674793 13526 ts_tablet_manager.cc:1918] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:27:36.674885 13526 log.cc:1196] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/4f926b8536e7461cb8799290a60617bb
I20260809 09:27:36.675148 13526 ts_tablet_manager.cc:1939] T 4f926b8536e7461cb8799290a60617bb P 1cfa683de1cd49c6ae7c48cab6d1f732: Deleting consensus metadata
I20260809 09:27:36.675364 12537 catalog_manager.cc:4841] TS 5440fdfa70904747a6f7886cc25be34f (127.9.102.2:33841): tablet 4f926b8536e7461cb8799290a60617bb (table test-table [id=4dd568398792464ca9a8c09d1e9bb6c9]) successfully deleted
I20260809 09:27:36.675769 12802 tablet_service.cc:1467] Processing CreateTablet for tablet 723e09e5151b41d7aeabc331699a9ef8 (DEFAULT_TABLE table=test-table [id=f928d959b5cc45aebb24d99a9b2eeb13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:36.675853 13041 tablet_service.cc:1467] Processing CreateTablet for tablet 723e09e5151b41d7aeabc331699a9ef8 (DEFAULT_TABLE table=test-table [id=f928d959b5cc45aebb24d99a9b2eeb13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:36.675998 12802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723e09e5151b41d7aeabc331699a9ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:36.676034 13041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723e09e5151b41d7aeabc331699a9ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:36.676942 13527 ts_tablet_manager.cc:1905] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:36.678097 13438 tablet_bootstrap.cc:492] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f: Bootstrap starting.
I20260809 09:27:36.678220 13440 tablet_bootstrap.cc:492] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26: Bootstrap starting.
I20260809 09:27:36.678985 13440 tablet_bootstrap.cc:654] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:36.678985 13438 tablet_bootstrap.cc:654] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:36.679421 13527 ts_tablet_manager.cc:1918] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:27:36.679500 13527 log.cc:1196] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/wal/wals/4f926b8536e7461cb8799290a60617bb
I20260809 09:27:36.679827 13527 ts_tablet_manager.cc:1939] T 4f926b8536e7461cb8799290a60617bb P e4ae233a060640b7a4875d0c90451d26: Deleting consensus metadata
I20260809 09:27:36.679987 12533 catalog_manager.cc:4841] TS 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289): tablet 4f926b8536e7461cb8799290a60617bb (table test-table [id=4dd568398792464ca9a8c09d1e9bb6c9]) successfully deleted
I20260809 09:27:36.680313 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 723e09e5151b41d7aeabc331699a9ef8 (DEFAULT_TABLE table=test-table [id=f928d959b5cc45aebb24d99a9b2eeb13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:36.680590 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723e09e5151b41d7aeabc331699a9ef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:36.680823 12539 catalog_manager.cc:4841] TS e4ae233a060640b7a4875d0c90451d26 (127.9.102.1:35483): tablet 4f926b8536e7461cb8799290a60617bb (table test-table [id=4dd568398792464ca9a8c09d1e9bb6c9]) successfully deleted
I20260809 09:27:36.682569 13439 tablet_bootstrap.cc:492] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732: Bootstrap starting.
I20260809 09:27:36.683418 13439 tablet_bootstrap.cc:654] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:36.684891 13440 tablet_bootstrap.cc:492] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26: No bootstrap required, opened a new log
I20260809 09:27:36.684892 13439 tablet_bootstrap.cc:492] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732: No bootstrap required, opened a new log
I20260809 09:27:36.684981 13439 ts_tablet_manager.cc:1397] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:36.684980 13440 ts_tablet_manager.cc:1397] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260809 09:27:36.685413 13439 raft_consensus.cc:348] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.685413 13440 raft_consensus.cc:348] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.685544 13440 raft_consensus.cc:374] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:36.685544 13439 raft_consensus.cc:374] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:36.685578 13439 raft_consensus.cc:729] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfa683de1cd49c6ae7c48cab6d1f732, State: Initialized, Role: FOLLOWER
I20260809 09:27:36.685578 13440 raft_consensus.cc:729] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ae233a060640b7a4875d0c90451d26, State: Initialized, Role: FOLLOWER
I20260809 09:27:36.685700 13440 consensus_queue.cc:260] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.685701 13439 consensus_queue.cc:260] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.685950 13439 ts_tablet_manager.cc:1428] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:27:36.685950 13440 ts_tablet_manager.cc:1428] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 09:27:36.687433 13438 tablet_bootstrap.cc:492] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f: No bootstrap required, opened a new log
I20260809 09:27:36.687537 13438 ts_tablet_manager.cc:1397] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260809 09:27:36.688025 13438 raft_consensus.cc:348] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.688169 13438 raft_consensus.cc:374] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:36.688218 13438 raft_consensus.cc:729] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5440fdfa70904747a6f7886cc25be34f, State: Initialized, Role: FOLLOWER
I20260809 09:27:36.688338 13438 consensus_queue.cc:260] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.688608 13438 ts_tablet_manager.cc:1428] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:27:36.992426 13447 raft_consensus.cc:482] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:27:36.992552 13447 raft_consensus.cc:504] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.993564 13447 leader_election.cc:288] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4ae233a060640b7a4875d0c90451d26 (127.9.102.1:35483), 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289)
I20260809 09:27:36.996757 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.993798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:52159 (local address 127.9.102.1:35483)
0809 09:27:36.993941 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.993946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.994045 (+ 99us) server_negotiation.cc:407] Connection header received
0809 09:27:36.994183 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.994186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.994236 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.994323 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:36.994546 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.994967 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.995865 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.996223 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.996344 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.996407 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.996581 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.996641 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:27:36.996771 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.993715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:35483 (local address 127.9.102.2:52159)
0809 09:27:36.993948 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:36.993964 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:36.994076 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:36.994334 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:36.994337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:36.994350 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:36.994444 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.994450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.995069 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.995074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.995747 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.995755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.996370 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.996373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.996455 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.996481 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 09:27:36.996564 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 09:27:36.996627 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260809 09:27:36.997018 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.993781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:34289 (local address 127.9.102.2:58609)
0809 09:27:36.994057 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:36.994070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:36.994177 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:36.994462 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:36.994464 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:36.994479 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:36.994572 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.994577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.995242 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.995245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.995900 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:36.995908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.996580 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.996583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:36.996718 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.996747 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 09:27:36.996829 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 09:27:36.996892 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":65,"threads_started":1}
I20260809 09:27:36.997045 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:36.994036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:58609 (local address 127.9.102.3:34289)
0809 09:27:36.994271 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:36.994274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:36.994285 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:27:36.994325 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:36.994327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:36.994359 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:36.994450 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:36.994689 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.995117 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.996031 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:36.996462 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:36.996575 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:36.996634 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:36.996857 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 09:27:36.996926 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260809 09:27:36.997323 12831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723e09e5151b41d7aeabc331699a9ef8" candidate_uuid: "5440fdfa70904747a6f7886cc25be34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ae233a060640b7a4875d0c90451d26" is_pre_election: true
I20260809 09:27:36.997454 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723e09e5151b41d7aeabc331699a9ef8" candidate_uuid: "5440fdfa70904747a6f7886cc25be34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" is_pre_election: true
I20260809 09:27:36.997560 12831 raft_consensus.cc:2454] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5440fdfa70904747a6f7886cc25be34f in term 0.
I20260809 09:27:36.997597 13323 raft_consensus.cc:2454] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5440fdfa70904747a6f7886cc25be34f in term 0.
I20260809 09:27:36.997907 12964 leader_election.cc:302] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfa683de1cd49c6ae7c48cab6d1f732, 5440fdfa70904747a6f7886cc25be34f; no voters:
I20260809 09:27:36.998138 13447 raft_consensus.cc:2781] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:27:36.998217 13447 raft_consensus.cc:482] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:27:36.998252 13447 raft_consensus.cc:3037] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:36.999177 13447 raft_consensus.cc:504] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:36.999491 13447 leader_election.cc:288] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [CANDIDATE]: Term 1 election: Requested vote from peers e4ae233a060640b7a4875d0c90451d26 (127.9.102.1:35483), 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289)
I20260809 09:27:36.999809 12831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723e09e5151b41d7aeabc331699a9ef8" candidate_uuid: "5440fdfa70904747a6f7886cc25be34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ae233a060640b7a4875d0c90451d26"
I20260809 09:27:36.999907 12831 raft_consensus.cc:3037] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:37.000226 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723e09e5151b41d7aeabc331699a9ef8" candidate_uuid: "5440fdfa70904747a6f7886cc25be34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732"
I20260809 09:27:37.000355 13323 raft_consensus.cc:3037] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:37.000927 12831 raft_consensus.cc:2454] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5440fdfa70904747a6f7886cc25be34f in term 1.
I20260809 09:27:37.001224 12967 leader_election.cc:302] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5440fdfa70904747a6f7886cc25be34f, e4ae233a060640b7a4875d0c90451d26; no voters:
I20260809 09:27:37.001410 13447 raft_consensus.cc:2781] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:37.001413 13323 raft_consensus.cc:2454] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5440fdfa70904747a6f7886cc25be34f in term 1.
I20260809 09:27:37.001652 13447 raft_consensus.cc:686] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 1 LEADER]: Becoming Leader. State: Replica: 5440fdfa70904747a6f7886cc25be34f, State: Running, Role: LEADER
I20260809 09:27:37.001965 13447 consensus_queue.cc:237] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:37.004809 12554 catalog_manager.cc:5495] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5440fdfa70904747a6f7886cc25be34f (127.9.102.2). New cstate: current_term: 1 leader_uuid: "5440fdfa70904747a6f7886cc25be34f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } health_report { overall_health: UNKNOWN } } }
I20260809 09:27:37.051694 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:37.048285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.9.102.2:33841)
0809 09:27:37.048478 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:37.048481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:37.048492 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:27:37.048577 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:37.048579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:37.048622 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:37.048711 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:37.048868 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:37.049615 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:37.050346 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:37.050549 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:37.050618 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:37.050900 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:37.051292 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:37.051372 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:37.051425 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 09:27:37.051482 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260809 09:27:37.059947 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:37.057506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:57929 (local address 127.9.102.1:35483)
0809 09:27:37.057655 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:37.057658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:37.057738 (+ 80us) server_negotiation.cc:407] Connection header received
0809 09:27:37.057836 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:37.057838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:37.057871 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:37.057924 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:37.058104 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:37.058488 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:37.059169 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:37.059341 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:37.059405 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:37.059646 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:37.059702 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:37.059753 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:37.059824 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 09:27:37.059860 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 09:27:37.059989 13549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:37.057413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:35483 (local address 127.9.102.2:57929)
0809 09:27:37.057667 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:37.057678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:37.057754 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:37.057941 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:37.057943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:37.057947 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 09:27:37.058019 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:37.058029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:37.058561 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:37.058563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:37.059076 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:37.059083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:37.059420 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:37.059423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:37.059482 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:37.059515 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 09:27:37.059769 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 09:27:37.059773 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 09:27:37.059819 (+ 46us) client_negotiation.cc:240] Negotiation successful
0809 09:27:37.059871 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":80,"threads_started":1}
I20260809 09:27:37.065086 13323 raft_consensus.cc:1261] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Refusing update from remote peer 5440fdfa70904747a6f7886cc25be34f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:37.065405 12831 raft_consensus.cc:1261] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Refusing update from remote peer 5440fdfa70904747a6f7886cc25be34f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:37.065552 13540 consensus_queue.cc:1035] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:37.065876 13447 consensus_queue.cc:1035] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:37.068823 13485 mvcc.cc:205] Tried to move back new op lower bound from 7316552323334651904 to 7316552323083063296. Current Snapshot: MvccSnapshot[applied={T|T < 7316552323334651904}]
I20260809 09:27:37.081346 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:37.078695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:49741 (local address 127.9.102.2:33841)
0809 09:27:37.078811 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:37.078815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:37.078824 (+ 9us) server_negotiation.cc:407] Connection header received
0809 09:27:37.078874 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:37.078877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:37.078916 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:37.079008 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:37.079236 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:37.079753 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:37.080516 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:37.080687 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:37.080785 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:37.080987 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:37.081046 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:37.081118 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:37.081211 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 09:27:37.081251 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 09:27:37.081395 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:37.078368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:33841 (local address 127.9.102.1:49741)
0809 09:27:37.078475 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:37.078489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:37.078569 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:37.079008 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:37.079010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:37.079016 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 09:27:37.079122 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:37.079130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:37.079849 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:37.079851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:37.080430 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:37.080439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:37.080786 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:37.080789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:37.080860 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:37.080887 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 09:27:37.081125 (+ 238us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 09:27:37.081132 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 09:27:37.081204 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 09:27:37.081276 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
W20260809 09:27:37.082104 13485 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 09:27:38.644169 12554 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:38.644302 12554 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:38.646332 12554 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3: Sending DeleteTablet for 3 replicas of tablet 723e09e5151b41d7aeabc331699a9ef8
I20260809 09:27:38.646800 12802 tablet_service.cc:1514] Processing DeleteTablet for tablet 723e09e5151b41d7aeabc331699a9ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:38 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:58902
I20260809 09:27:38.646865 13041 tablet_service.cc:1514] Processing DeleteTablet for tablet 723e09e5151b41d7aeabc331699a9ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:38 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:43222
I20260809 09:27:38.646917 13289 tablet_service.cc:1514] Processing DeleteTablet for tablet 723e09e5151b41d7aeabc331699a9ef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:38 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:56444
I20260809 09:27:38.647138 13598 tablet_replica.cc:330] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f: stopping tablet replica
I20260809 09:27:38.647182 13599 tablet_replica.cc:330] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732: stopping tablet replica
I20260809 09:27:38.647254 13598 raft_consensus.cc:2229] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:27:38.647307 13599 raft_consensus.cc:2229] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:38.647413 13599 raft_consensus.cc:2258] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:38.647416 13597 tablet_replica.cc:330] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26: stopping tablet replica
I20260809 09:27:38.647543 13598 raft_consensus.cc:2258] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:38.647688 13597 raft_consensus.cc:2229] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:38.647830 13597 raft_consensus.cc:2258] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:38.648187 13598 ts_tablet_manager.cc:1905] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:38.648491 13597 ts_tablet_manager.cc:1905] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:38.650904 13599 ts_tablet_manager.cc:1905] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:38.651191 13598 ts_tablet_manager.cc:1918] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 09:27:38.651278 13598 log.cc:1196] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/723e09e5151b41d7aeabc331699a9ef8
I20260809 09:27:38.651336 13597 ts_tablet_manager.cc:1918] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 09:27:38.651412 13597 log.cc:1196] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/wal/wals/723e09e5151b41d7aeabc331699a9ef8
I20260809 09:27:38.651583 13598 ts_tablet_manager.cc:1939] T 723e09e5151b41d7aeabc331699a9ef8 P 5440fdfa70904747a6f7886cc25be34f: Deleting consensus metadata
I20260809 09:27:38.651690 13597 ts_tablet_manager.cc:1939] T 723e09e5151b41d7aeabc331699a9ef8 P e4ae233a060640b7a4875d0c90451d26: Deleting consensus metadata
I20260809 09:27:38.652426 12537 catalog_manager.cc:4841] TS 5440fdfa70904747a6f7886cc25be34f (127.9.102.2:33841): tablet 723e09e5151b41d7aeabc331699a9ef8 (table test-table [id=f928d959b5cc45aebb24d99a9b2eeb13]) successfully deleted
I20260809 09:27:38.652462 12539 catalog_manager.cc:4841] TS e4ae233a060640b7a4875d0c90451d26 (127.9.102.1:35483): tablet 723e09e5151b41d7aeabc331699a9ef8 (table test-table [id=f928d959b5cc45aebb24d99a9b2eeb13]) successfully deleted
I20260809 09:27:38.653568 13599 ts_tablet_manager.cc:1918] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 09:27:38.653632 13599 log.cc:1196] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/723e09e5151b41d7aeabc331699a9ef8
I20260809 09:27:38.653915 13599 ts_tablet_manager.cc:1939] T 723e09e5151b41d7aeabc331699a9ef8 P 1cfa683de1cd49c6ae7c48cab6d1f732: Deleting consensus metadata
I20260809 09:27:38.654525 12533 catalog_manager.cc:4841] TS 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289): tablet 723e09e5151b41d7aeabc331699a9ef8 (table test-table [id=f928d959b5cc45aebb24d99a9b2eeb13]) successfully deleted
Aug 09 09:27:38 dist-test-slave-7vjv krb5kdc[12486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267658, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 09:27:38 dist-test-slave-7vjv krb5kdc[12486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267658, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:38.676285 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:38.665869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.9.102.62:42791)
0809 09:27:38.666130 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:38.666136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:38.666148 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:27:38.666251 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:38.666253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:38.666289 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:38.666354 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:38.666905 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:38.667364 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:38.668075 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:38.668337 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:38.668409 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:38.673861 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:38.673863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:38.673865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:38.673884 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:38.674990 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:38.675317 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:38.675319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:38.675320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:38.675385 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:38.675597 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:38.675599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:38.675601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:38.675769 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:38.675861 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:38.676025 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 09:27:38.676168 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260809 09:27:38.678184 12554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:27:38.678571 12554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:27:38.682277 12802 tablet_service.cc:1467] Processing CreateTablet for tablet 35fc7750e75e4f0b9f87c9571649144e (DEFAULT_TABLE table=test-table [id=9e1cf7364add40bd932ea18c9c7f7841]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:38.682425 13041 tablet_service.cc:1467] Processing CreateTablet for tablet 35fc7750e75e4f0b9f87c9571649144e (DEFAULT_TABLE table=test-table [id=9e1cf7364add40bd932ea18c9c7f7841]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:38.682544 12802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fc7750e75e4f0b9f87c9571649144e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:38.682695 13041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fc7750e75e4f0b9f87c9571649144e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:38.682705 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 35fc7750e75e4f0b9f87c9571649144e (DEFAULT_TABLE table=test-table [id=9e1cf7364add40bd932ea18c9c7f7841]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:38.682911 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35fc7750e75e4f0b9f87c9571649144e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:38.684422 13614 tablet_bootstrap.cc:492] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26: Bootstrap starting.
I20260809 09:27:38.684900 13619 tablet_bootstrap.cc:492] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f: Bootstrap starting.
I20260809 09:27:38.685091 13621 tablet_bootstrap.cc:492] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732: Bootstrap starting.
I20260809 09:27:38.685585 13614 tablet_bootstrap.cc:654] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:38.685941 13621 tablet_bootstrap.cc:654] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:38.686679 13614 tablet_bootstrap.cc:492] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26: No bootstrap required, opened a new log
I20260809 09:27:38.686766 13614 ts_tablet_manager.cc:1397] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:38.686944 13621 tablet_bootstrap.cc:492] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732: No bootstrap required, opened a new log
I20260809 09:27:38.687032 13621 ts_tablet_manager.cc:1397] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:38.687103 13619 tablet_bootstrap.cc:654] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:38.687160 13614 raft_consensus.cc:348] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.687266 13614 raft_consensus.cc:374] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:38.687304 13614 raft_consensus.cc:729] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ae233a060640b7a4875d0c90451d26, State: Initialized, Role: FOLLOWER
I20260809 09:27:38.687417 13621 raft_consensus.cc:348] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.687453 13614 consensus_queue.cc:260] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.687508 13621 raft_consensus.cc:374] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:38.687552 13621 raft_consensus.cc:729] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfa683de1cd49c6ae7c48cab6d1f732, State: Initialized, Role: FOLLOWER
I20260809 09:27:38.687671 13621 consensus_queue.cc:260] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.688014 13621 ts_tablet_manager.cc:1428] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:27:38.688239 13619 tablet_bootstrap.cc:492] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f: No bootstrap required, opened a new log
I20260809 09:27:38.688314 13619 ts_tablet_manager.cc:1397] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 09:27:38.688663 13619 raft_consensus.cc:348] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.688781 13619 raft_consensus.cc:374] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:38.688817 13619 raft_consensus.cc:729] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5440fdfa70904747a6f7886cc25be34f, State: Initialized, Role: FOLLOWER
I20260809 09:27:38.688944 13619 consensus_queue.cc:260] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.689205 13619 ts_tablet_manager.cc:1428] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 09:27:38.689599 13614 ts_tablet_manager.cc:1428] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260809 09:27:38.698163 13591 raft_consensus.cc:482] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:27:38.698251 13591 raft_consensus.cc:504] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.698546 13591 leader_election.cc:288] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4ae233a060640b7a4875d0c90451d26 (127.9.102.1:35483), 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289)
I20260809 09:27:38.698860 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fc7750e75e4f0b9f87c9571649144e" candidate_uuid: "5440fdfa70904747a6f7886cc25be34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" is_pre_election: true
I20260809 09:27:38.698871 12831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fc7750e75e4f0b9f87c9571649144e" candidate_uuid: "5440fdfa70904747a6f7886cc25be34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ae233a060640b7a4875d0c90451d26" is_pre_election: true
I20260809 09:27:38.698971 13323 raft_consensus.cc:2454] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5440fdfa70904747a6f7886cc25be34f in term 0.
I20260809 09:27:38.698997 12831 raft_consensus.cc:2454] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5440fdfa70904747a6f7886cc25be34f in term 0.
I20260809 09:27:38.699258 12964 leader_election.cc:302] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfa683de1cd49c6ae7c48cab6d1f732, 5440fdfa70904747a6f7886cc25be34f; no voters:
I20260809 09:27:38.699440 13591 raft_consensus.cc:2781] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:27:38.699512 13591 raft_consensus.cc:482] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:27:38.699556 13591 raft_consensus.cc:3037] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:38.700496 13591 raft_consensus.cc:504] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.700819 13591 leader_election.cc:288] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [CANDIDATE]: Term 1 election: Requested vote from peers e4ae233a060640b7a4875d0c90451d26 (127.9.102.1:35483), 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289)
I20260809 09:27:38.701097 12831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fc7750e75e4f0b9f87c9571649144e" candidate_uuid: "5440fdfa70904747a6f7886cc25be34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4ae233a060640b7a4875d0c90451d26"
I20260809 09:27:38.701144 13323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35fc7750e75e4f0b9f87c9571649144e" candidate_uuid: "5440fdfa70904747a6f7886cc25be34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732"
I20260809 09:27:38.701217 12831 raft_consensus.cc:3037] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:38.701280 13323 raft_consensus.cc:3037] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:38.701970 13323 raft_consensus.cc:2454] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5440fdfa70904747a6f7886cc25be34f in term 1.
I20260809 09:27:38.702024 12831 raft_consensus.cc:2454] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5440fdfa70904747a6f7886cc25be34f in term 1.
I20260809 09:27:38.702294 12964 leader_election.cc:302] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfa683de1cd49c6ae7c48cab6d1f732, 5440fdfa70904747a6f7886cc25be34f; no voters:
I20260809 09:27:38.702471 13591 raft_consensus.cc:2781] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:38.702584 13591 raft_consensus.cc:686] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 1 LEADER]: Becoming Leader. State: Replica: 5440fdfa70904747a6f7886cc25be34f, State: Running, Role: LEADER
I20260809 09:27:38.702692 13591 consensus_queue.cc:237] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.703940 12554 catalog_manager.cc:5495] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5440fdfa70904747a6f7886cc25be34f (127.9.102.2). New cstate: current_term: 1 leader_uuid: "5440fdfa70904747a6f7886cc25be34f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } health_report { overall_health: UNKNOWN } } }
I20260809 09:27:38.715942 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:38.713013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.9.102.2:33841)
0809 09:27:38.713277 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:38.713293 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:38.713304 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:27:38.713337 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:38.713339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:38.713374 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:38.713464 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:38.713737 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:38.714221 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:38.714991 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:38.715200 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:38.715301 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:38.715544 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:38.715604 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:38.715675 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:38.715753 (+ 78us) server_negotiation.cc:299] Negotiation successful
0809 09:27:38.715812 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
I20260809 09:27:38.721367 12831 raft_consensus.cc:1261] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Refusing update from remote peer 5440fdfa70904747a6f7886cc25be34f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:38.721374 13323 raft_consensus.cc:1261] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Refusing update from remote peer 5440fdfa70904747a6f7886cc25be34f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:38.721736 13591 consensus_queue.cc:1035] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:38.721853 13586 consensus_queue.cc:1035] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:38.751564 12552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:38.751708 12552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:38.753793 12552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3: Sending DeleteTablet for 3 replicas of tablet 35fc7750e75e4f0b9f87c9571649144e
I20260809 09:27:38.754189 12802 tablet_service.cc:1514] Processing DeleteTablet for tablet 35fc7750e75e4f0b9f87c9571649144e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:38 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:58902
I20260809 09:27:38.754362 13597 tablet_replica.cc:330] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26: stopping tablet replica
I20260809 09:27:38.754477 13597 raft_consensus.cc:2229] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:38.754557 13597 raft_consensus.cc:2258] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:38.754717 13289 tablet_service.cc:1514] Processing DeleteTablet for tablet 35fc7750e75e4f0b9f87c9571649144e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:38 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:56444
I20260809 09:27:38.754869 13599 tablet_replica.cc:330] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732: stopping tablet replica
I20260809 09:27:38.754987 13041 tablet_service.cc:1514] Processing DeleteTablet for tablet 35fc7750e75e4f0b9f87c9571649144e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:38 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:43222
I20260809 09:27:38.754976 13599 raft_consensus.cc:2229] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:38.755105 13599 raft_consensus.cc:2258] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:38.755134 13598 tablet_replica.cc:330] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f: stopping tablet replica
I20260809 09:27:38.755239 13598 raft_consensus.cc:2229] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:27:38.755417 13598 raft_consensus.cc:2258] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:38.755714 13599 ts_tablet_manager.cc:1905] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:38.756013 13598 ts_tablet_manager.cc:1905] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:38.758319 13599 ts_tablet_manager.cc:1918] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:27:38.758374 13598 ts_tablet_manager.cc:1918] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:27:38.758407 13599 log.cc:1196] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/35fc7750e75e4f0b9f87c9571649144e
I20260809 09:27:38.758436 13598 log.cc:1196] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/35fc7750e75e4f0b9f87c9571649144e
I20260809 09:27:38.758685 13599 ts_tablet_manager.cc:1939] T 35fc7750e75e4f0b9f87c9571649144e P 1cfa683de1cd49c6ae7c48cab6d1f732: Deleting consensus metadata
I20260809 09:27:38.758685 13598 ts_tablet_manager.cc:1939] T 35fc7750e75e4f0b9f87c9571649144e P 5440fdfa70904747a6f7886cc25be34f: Deleting consensus metadata
I20260809 09:27:38.759316 12533 catalog_manager.cc:4841] TS 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289): tablet 35fc7750e75e4f0b9f87c9571649144e (table test-table [id=9e1cf7364add40bd932ea18c9c7f7841]) successfully deleted
I20260809 09:27:38.759366 12552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:27:38.759789 12552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:27:38.759472 12537 catalog_manager.cc:4841] TS 5440fdfa70904747a6f7886cc25be34f (127.9.102.2:33841): tablet 35fc7750e75e4f0b9f87c9571649144e (table test-table [id=9e1cf7364add40bd932ea18c9c7f7841]) successfully deleted
I20260809 09:27:38.760694 13597 ts_tablet_manager.cc:1905] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:38.763154 13597 ts_tablet_manager.cc:1918] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:27:38.763233 13597 log.cc:1196] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-0/wal/wals/35fc7750e75e4f0b9f87c9571649144e
I20260809 09:27:38.763494 13597 ts_tablet_manager.cc:1939] T 35fc7750e75e4f0b9f87c9571649144e P e4ae233a060640b7a4875d0c90451d26: Deleting consensus metadata
I20260809 09:27:38.764120 12802 tablet_service.cc:1467] Processing CreateTablet for tablet 18a366306e8341f8bc89888860e49980 (DEFAULT_TABLE table=test-table [id=ddd5fb315dd8445ea77e54724dca6a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:38.764170 12539 catalog_manager.cc:4841] TS e4ae233a060640b7a4875d0c90451d26 (127.9.102.1:35483): tablet 35fc7750e75e4f0b9f87c9571649144e (table test-table [id=9e1cf7364add40bd932ea18c9c7f7841]) successfully deleted
I20260809 09:27:38.764274 13041 tablet_service.cc:1467] Processing CreateTablet for tablet 18a366306e8341f8bc89888860e49980 (DEFAULT_TABLE table=test-table [id=ddd5fb315dd8445ea77e54724dca6a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:38.764433 12802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a366306e8341f8bc89888860e49980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:38.764547 13041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a366306e8341f8bc89888860e49980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:38.764581 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 18a366306e8341f8bc89888860e49980 (DEFAULT_TABLE table=test-table [id=ddd5fb315dd8445ea77e54724dca6a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:38.764806 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a366306e8341f8bc89888860e49980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:38.766350 13614 tablet_bootstrap.cc:492] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26: Bootstrap starting.
I20260809 09:27:38.766734 13619 tablet_bootstrap.cc:492] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f: Bootstrap starting.
I20260809 09:27:38.767007 13621 tablet_bootstrap.cc:492] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732: Bootstrap starting.
I20260809 09:27:38.767167 13614 tablet_bootstrap.cc:654] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:38.767570 13619 tablet_bootstrap.cc:654] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:38.768383 13621 tablet_bootstrap.cc:654] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:38.768404 13614 tablet_bootstrap.cc:492] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26: No bootstrap required, opened a new log
I20260809 09:27:38.768478 13614 ts_tablet_manager.cc:1397] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:38.768687 13619 tablet_bootstrap.cc:492] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f: No bootstrap required, opened a new log
I20260809 09:27:38.768777 13619 ts_tablet_manager.cc:1397] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:38.768858 13614 raft_consensus.cc:348] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.768955 13614 raft_consensus.cc:374] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:38.768990 13614 raft_consensus.cc:729] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4ae233a060640b7a4875d0c90451d26, State: Initialized, Role: FOLLOWER
I20260809 09:27:38.769084 13614 consensus_queue.cc:260] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.769116 13619 raft_consensus.cc:348] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.769193 13619 raft_consensus.cc:374] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:38.769234 13619 raft_consensus.cc:729] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5440fdfa70904747a6f7886cc25be34f, State: Initialized, Role: FOLLOWER
I20260809 09:27:38.769330 13614 ts_tablet_manager.cc:1428] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 09:27:38.769335 13619 consensus_queue.cc:260] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.769577 13619 ts_tablet_manager.cc:1428] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:27:38.770303 13621 tablet_bootstrap.cc:492] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732: No bootstrap required, opened a new log
I20260809 09:27:38.770399 13621 ts_tablet_manager.cc:1397] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260809 09:27:38.770776 13621 raft_consensus.cc:348] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.770886 13621 raft_consensus.cc:374] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:38.770931 13621 raft_consensus.cc:729] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfa683de1cd49c6ae7c48cab6d1f732, State: Initialized, Role: FOLLOWER
I20260809 09:27:38.771045 13621 consensus_queue.cc:260] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.771315 13621 ts_tablet_manager.cc:1428] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 09:27:38.786453 13559 raft_consensus.cc:482] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:27:38.786545 13559 raft_consensus.cc:504] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.786872 13559 leader_election.cc:288] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5440fdfa70904747a6f7886cc25be34f (127.9.102.2:33841), 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289)
I20260809 09:27:38.787137 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a366306e8341f8bc89888860e49980" candidate_uuid: "e4ae233a060640b7a4875d0c90451d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5440fdfa70904747a6f7886cc25be34f" is_pre_election: true
I20260809 09:27:38.787231 13071 raft_consensus.cc:2454] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ae233a060640b7a4875d0c90451d26 in term 0.
I20260809 09:27:38.787235 13321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a366306e8341f8bc89888860e49980" candidate_uuid: "e4ae233a060640b7a4875d0c90451d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" is_pre_election: true
I20260809 09:27:38.787382 13321 raft_consensus.cc:2454] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4ae233a060640b7a4875d0c90451d26 in term 0.
I20260809 09:27:38.787546 12717 leader_election.cc:302] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5440fdfa70904747a6f7886cc25be34f, e4ae233a060640b7a4875d0c90451d26; no voters:
I20260809 09:27:38.787725 13559 raft_consensus.cc:2781] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:27:38.787794 13559 raft_consensus.cc:482] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:27:38.787861 13559 raft_consensus.cc:3037] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:38.788658 13559 raft_consensus.cc:504] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.788973 13559 leader_election.cc:288] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [CANDIDATE]: Term 1 election: Requested vote from peers 5440fdfa70904747a6f7886cc25be34f (127.9.102.2:33841), 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289)
I20260809 09:27:38.789224 13071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a366306e8341f8bc89888860e49980" candidate_uuid: "e4ae233a060640b7a4875d0c90451d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5440fdfa70904747a6f7886cc25be34f"
I20260809 09:27:38.789283 13321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a366306e8341f8bc89888860e49980" candidate_uuid: "e4ae233a060640b7a4875d0c90451d26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732"
I20260809 09:27:38.789340 13071 raft_consensus.cc:3037] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:38.789400 13321 raft_consensus.cc:3037] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:38.790113 13071 raft_consensus.cc:2454] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ae233a060640b7a4875d0c90451d26 in term 1.
I20260809 09:27:38.790366 13321 raft_consensus.cc:2454] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4ae233a060640b7a4875d0c90451d26 in term 1.
I20260809 09:27:38.790443 12717 leader_election.cc:302] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5440fdfa70904747a6f7886cc25be34f, e4ae233a060640b7a4875d0c90451d26; no voters:
I20260809 09:27:38.790642 13559 raft_consensus.cc:2781] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:38.790737 13559 raft_consensus.cc:686] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [term 1 LEADER]: Becoming Leader. State: Replica: e4ae233a060640b7a4875d0c90451d26, State: Running, Role: LEADER
I20260809 09:27:38.790855 13559 consensus_queue.cc:237] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } }
I20260809 09:27:38.792296 12551 catalog_manager.cc:5495] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4ae233a060640b7a4875d0c90451d26 (127.9.102.1). New cstate: current_term: 1 leader_uuid: "e4ae233a060640b7a4875d0c90451d26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4ae233a060640b7a4875d0c90451d26" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 } health_report { overall_health: UNKNOWN } } }
I20260809 09:27:38.806767 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:38.804344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.9.102.1:35483)
0809 09:27:38.804577 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:38.804583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:38.804593 (+ 10us) server_negotiation.cc:407] Connection header received
0809 09:27:38.804627 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:38.804629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:38.804661 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:38.804725 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:38.804886 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:38.805347 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:38.806024 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:38.806215 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:38.806285 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:38.806460 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:38.806513 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:38.806562 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:38.806641 (+ 79us) server_negotiation.cc:299] Negotiation successful
0809 09:27:38.806677 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":107,"threads_started":1}
I20260809 09:27:38.811838 13071 raft_consensus.cc:1261] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Refusing update from remote peer e4ae233a060640b7a4875d0c90451d26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:38.811869 13321 raft_consensus.cc:1261] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Refusing update from remote peer e4ae233a060640b7a4875d0c90451d26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:38.812168 13559 consensus_queue.cc:1035] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5440fdfa70904747a6f7886cc25be34f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 33841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:38.812297 13570 consensus_queue.cc:1035] T 18a366306e8341f8bc89888860e49980 P e4ae233a060640b7a4875d0c90451d26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfa683de1cd49c6ae7c48cab6d1f732" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 34289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 09:27:38.823930 13557 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 09:27:40.369100 12554 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:40.369232 12554 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:40.371393 12554 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 213728ff271b45cbbf2e1696e914f6b3: Sending DeleteTablet for 3 replicas of tablet 18a366306e8341f8bc89888860e49980
I20260809 09:27:40.371898 13041 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a366306e8341f8bc89888860e49980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:40 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:43222
I20260809 09:27:40.371970 12800 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a366306e8341f8bc89888860e49980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:40 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:58902
I20260809 09:27:40.372069 13289 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a366306e8341f8bc89888860e49980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:40 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:56444
I20260809 09:27:40.372252 13684 tablet_replica.cc:330] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f: stopping tablet replica
I20260809 09:27:40.372380 13684 raft_consensus.cc:2229] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:40.372409 13693 tablet_replica.cc:330] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732: stopping tablet replica
I20260809 09:27:40.372481 13684 raft_consensus.cc:2258] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:40.372529 13693 raft_consensus.cc:2229] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:40.372525 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 12686
I20260809 09:27:40.396394 13693 raft_consensus.cc:2258] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:40.397924 13693 ts_tablet_manager.cc:1905] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:40.398316 13684 ts_tablet_manager.cc:1905] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:40.400342 13693 ts_tablet_manager.cc:1918] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 09:27:40.400452 13693 log.cc:1196] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/18a366306e8341f8bc89888860e49980
I20260809 09:27:40.400796 13693 ts_tablet_manager.cc:1939] T 18a366306e8341f8bc89888860e49980 P 1cfa683de1cd49c6ae7c48cab6d1f732: Deleting consensus metadata
I20260809 09:27:40.401453 12533 catalog_manager.cc:4841] TS 1cfa683de1cd49c6ae7c48cab6d1f732 (127.9.102.3:34289): tablet 18a366306e8341f8bc89888860e49980 (table test-table [id=ddd5fb315dd8445ea77e54724dca6a19]) successfully deleted
I20260809 09:27:40.401721 13684 ts_tablet_manager.cc:1918] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 09:27:40.401804 13684 log.cc:1196] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/18a366306e8341f8bc89888860e49980
I20260809 09:27:40.402062 13684 ts_tablet_manager.cc:1939] T 18a366306e8341f8bc89888860e49980 P 5440fdfa70904747a6f7886cc25be34f: Deleting consensus metadata
I20260809 09:27:40.402727 12537 catalog_manager.cc:4841] TS 5440fdfa70904747a6f7886cc25be34f (127.9.102.2:33841): tablet 18a366306e8341f8bc89888860e49980 (table test-table [id=ddd5fb315dd8445ea77e54724dca6a19]) successfully deleted
W20260809 09:27:40.405838 12539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:35483 (error 108)
I20260809 09:27:40.405937 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 12927
I20260809 09:27:40.412721 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:40.412352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:35483 (local address 127.0.0.1:48200)
0809 09:27:40.412570 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:40.412646 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
W20260809 09:27:40.412891 12539 catalog_manager.cc:4584] TS e4ae233a060640b7a4875d0c90451d26 (127.9.102.1:35483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18a366306e8341f8bc89888860e49980: Network error: Client connection negotiation failed: client connection to 127.9.102.1:35483: connect: Connection refused (error 111)
I20260809 09:27:40.417480 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 13179
I20260809 09:27:40.429793 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 12513
2026-08-09T09:27:40Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6378 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:27:40 dist-test-slave-7vjv krb5kdc[13707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:27:40 dist-test-slave-7vjv krb5kdc[13707](info): set up 2 sockets
Aug 09 09:27:40 dist-test-slave-7vjv krb5kdc[13707](info): commencing operation
krb5kdc: starting...
W20260809 09:27:41.595574 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:27:41 dist-test-slave-7vjv krb5kdc[13707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:27:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:27:41.704836 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:42513
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:42513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:41.802605 13727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:41.802834 13727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:41.802896 13727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:41.806370 13727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:41.806438 13727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:41.806473 13727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:41.806522 13727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:41.806552 13727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:41.811097 13727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:42513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:42513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:41.811935 13727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:41.812810 13727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:41.820185 13736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:41.820185 13735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:41.820343 13727 server_base.cc:1034] running on GCE node
W20260809 09:27:41.820472 13738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:41.820950 13727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:41.821812 13727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:41.822990 13727 hybrid_clock.cc:648] HybridClock initialized: now 1786267661822975 us; error 26 us; skew 500 ppm
Aug 09 09:27:41 dist-test-slave-7vjv krb5kdc[13707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267661, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:41.825448 13727 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:27:41.826205 13727 webserver.cc:460] Webserver started at http://127.9.102.62:43351/ using document root <none> and password file <none>
I20260809 09:27:41.826666 13727 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:41.826750 13727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:41.826952 13727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:41.828768 13727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "8919c710bcf74748a16e089dd664fd96"
format_stamp: "Formatted at 2026-08-09 09:27:41 on dist-test-slave-7vjv"
server_key: "cb9de95694ca18f0557efd1b4f680f64"
server_key_iv: "07c1644ae1599fdd8c51afc6cb210836"
server_key_version: "encryptionkey@0"
I20260809 09:27:41.829257 13727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "8919c710bcf74748a16e089dd664fd96"
format_stamp: "Formatted at 2026-08-09 09:27:41 on dist-test-slave-7vjv"
server_key: "cb9de95694ca18f0557efd1b4f680f64"
server_key_iv: "07c1644ae1599fdd8c51afc6cb210836"
server_key_version: "encryptionkey@0"
I20260809 09:27:41.832408 13727 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:27:41.834411 13752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:41.835491 13727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:41.835628 13727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "8919c710bcf74748a16e089dd664fd96"
format_stamp: "Formatted at 2026-08-09 09:27:41 on dist-test-slave-7vjv"
server_key: "cb9de95694ca18f0557efd1b4f680f64"
server_key_iv: "07c1644ae1599fdd8c51afc6cb210836"
server_key_version: "encryptionkey@0"
I20260809 09:27:41.835731 13727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:41.861058 13727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:41.863304 13727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:41.863469 13727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:41.869379 13727 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:42513
I20260809 09:27:41.869413 13841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:42513 every 8 connection(s)
I20260809 09:27:41.870173 13727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:41.872478 13843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:41.877228 13843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96: Bootstrap starting.
I20260809 09:27:41.879329 13843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:41.879869 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 13727
I20260809 09:27:41.879976 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:41.880199 13843 log.cc:824] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:41.880260 9624 external_mini_cluster.cc:1428] Setting key e1b7c37cbee032da7f54d7316542254e
I20260809 09:27:41.882012 13843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96: No bootstrap required, opened a new log
I20260809 09:27:41.885705 13843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8919c710bcf74748a16e089dd664fd96" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42513 } }
I20260809 09:27:41.885929 13843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:41.885982 13843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8919c710bcf74748a16e089dd664fd96, State: Initialized, Role: FOLLOWER
I20260809 09:27:41.886412 13843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8919c710bcf74748a16e089dd664fd96" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42513 } }
I20260809 09:27:41.886579 13843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:41.886653 13843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:41.886770 13843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:41.887588 13843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8919c710bcf74748a16e089dd664fd96" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42513 } }
I20260809 09:27:41.887876 13843 leader_election.cc:302] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8919c710bcf74748a16e089dd664fd96; no voters:
I20260809 09:27:41.888234 13843 leader_election.cc:288] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:41.888329 13849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:41.888573 13849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [term 1 LEADER]: Becoming Leader. State: Replica: 8919c710bcf74748a16e089dd664fd96, State: Running, Role: LEADER
Aug 09 09:27:41 dist-test-slave-7vjv krb5kdc[13707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:41.888845 13849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8919c710bcf74748a16e089dd664fd96" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42513 } }
I20260809 09:27:41.889389 13843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:41.890595 13850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8919c710bcf74748a16e089dd664fd96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8919c710bcf74748a16e089dd664fd96" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42513 } } }
I20260809 09:27:41.890728 13850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:41.891012 13855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8919c710bcf74748a16e089dd664fd96. Latest consensus state: current_term: 1 leader_uuid: "8919c710bcf74748a16e089dd664fd96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8919c710bcf74748a16e089dd664fd96" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42513 } } }
I20260809 09:27:41.891116 13855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:41.891387 13867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:41.894246 13867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:41.894979 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:41.881231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.9.102.62:42513)
0809 09:27:41.881618 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:41.881634 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:41.881669 (+ 35us) server_negotiation.cc:407] Connection header received
0809 09:27:41.882348 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:41.882368 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:41.882693 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:41.882987 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:41.883555 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:41.884971 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:41.885737 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:41.886359 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:41.886513 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:41.889249 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:41.889254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:41.889268 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:41.889307 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:41.892215 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:41.892683 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:41.892690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:41.892695 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:41.892816 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:41.893113 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:41.893118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:41.893120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:41.893543 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:41.893699 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:41.893968 (+ 269us) server_negotiation.cc:299] Negotiation successful
0809 09:27:41.894204 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20260809 09:27:41.898789 13867 catalog_manager.cc:1347] Generated new cluster ID: e79ab18db3a3436badeaaab3dbbfee85
I20260809 09:27:41.898845 13867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:41.922731 13867 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:41.923480 13867 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:41.927615 13867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8919c710bcf74748a16e089dd664fd96: Generated new TSK 0
I20260809 09:27:41.928189 13867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:27:41.955421 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:42513
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:42.058605 13884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:42.058818 13884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:42.058871 13884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:42.062575 13884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:42.062620 13884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:42.062667 13884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:27:42.067138 13884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:42513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:42.067965 13884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:42.068807 13884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:42.077248 13892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:42.077310 13884 server_base.cc:1034] running on GCE node
W20260809 09:27:42.077260 13897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:42.077251 13891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:42.077821 13884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:42.078359 13884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:42.079536 13884 hybrid_clock.cc:648] HybridClock initialized: now 1786267662079526 us; error 34 us; skew 500 ppm
Aug 09 09:27:42 dist-test-slave-7vjv krb5kdc[13707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267662, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:42.082122 13884 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:27:42.082850 13884 webserver.cc:460] Webserver started at http://127.9.102.1:41013/ using document root <none> and password file <none>
I20260809 09:27:42.083297 13884 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:42.083350 13884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:42.083490 13884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:42.085207 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "357f1d3bcb684d55a6ffb12b4df7dcf4"
format_stamp: "Formatted at 2026-08-09 09:27:42 on dist-test-slave-7vjv"
server_key: "bda6bdf938b6db8fd067de4934fc4c44"
server_key_iv: "cd06389ab98eb13846a0298d97ddf1fe"
server_key_version: "encryptionkey@0"
I20260809 09:27:42.085668 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "357f1d3bcb684d55a6ffb12b4df7dcf4"
format_stamp: "Formatted at 2026-08-09 09:27:42 on dist-test-slave-7vjv"
server_key: "bda6bdf938b6db8fd067de4934fc4c44"
server_key_iv: "cd06389ab98eb13846a0298d97ddf1fe"
server_key_version: "encryptionkey@0"
I20260809 09:27:42.088701 13884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:27:42.090903 13913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:42.091988 13884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:42.092077 13884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "357f1d3bcb684d55a6ffb12b4df7dcf4"
format_stamp: "Formatted at 2026-08-09 09:27:42 on dist-test-slave-7vjv"
server_key: "bda6bdf938b6db8fd067de4934fc4c44"
server_key_iv: "cd06389ab98eb13846a0298d97ddf1fe"
server_key_version: "encryptionkey@0"
I20260809 09:27:42.092203 13884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:42.104364 13884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:42.106626 13884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:42.106760 13884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:42.107153 13884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:42.107771 13884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:42.107812 13884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:42.107843 13884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:42.107861 13884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:42.116077 13884 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:44007
I20260809 09:27:42.116155 14122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:44007 every 8 connection(s)
I20260809 09:27:42.116874 13884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:27:42.120775 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 13884
I20260809 09:27:42.120899 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:27:42.121145 9624 external_mini_cluster.cc:1428] Setting key 978c97d3129cf1a5fa4df4631ed6666e
Aug 09 09:27:42 dist-test-slave-7vjv krb5kdc[13707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267662, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:42.129638 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:42.118341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:32955 (local address 127.9.102.62:42513)
0809 09:27:42.118455 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:42.118458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:42.119125 (+ 667us) server_negotiation.cc:407] Connection header received
0809 09:27:42.120327 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:42.120330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:42.120373 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:42.120438 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:42.121629 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:42.122301 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:42.123305 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:42.123523 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:42.123614 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:42.126013 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:42.126016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:42.126018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:42.126040 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:42.127581 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:42.128021 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:42.128026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:42.128029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:42.128153 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:42.128510 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:42.128516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:42.128520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:42.128781 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:42.128898 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:42.129326 (+ 428us) server_negotiation.cc:299] Negotiation successful
0809 09:27:42.129474 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 09:27:42.130131 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:42.118625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:42513 (local address 127.9.102.1:32955)
0809 09:27:42.118999 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:42.119035 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:42.120126 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:42.120614 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:42.120621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:42.120992 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:42.121486 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:42.121499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:42.122427 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:42.122430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:42.123187 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:42.123196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:42.123657 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:42.123661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:42.123800 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:42.124371 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:42.124393 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:42.125860 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:42.127690 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:42.127694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:42.127703 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:42.127918 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:42.128261 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:42.128264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:42.128265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:42.128409 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:42.128893 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:42.128898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:42.129095 (+ 197us) client_negotiation.cc:769] Sending connection context
0809 09:27:42.129233 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 09:27:42.129444 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260809 09:27:42.131193 14123 heartbeater.cc:347] Connected to a master server at 127.9.102.62:42513
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:27:42.131409 14123 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:42.131876 14123 heartbeater.cc:511] Master 127.9.102.62:42513 requested a full tablet report, sending...
I20260809 09:27:42.133230 13784 ts_manager.cc:194] Registered new tserver with Master: 357f1d3bcb684d55a6ffb12b4df7dcf4 (127.9.102.1:44007)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
I20260809 09:27:42.134380 13784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:32955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:27:42.146878 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:42513
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:42.242455 14130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:42.242676 14130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:42.242738 14130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:42.246168 14130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:42.246249 14130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:42.246356 14130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:27:42.250797 14130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:42513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:42.251583 14130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:42.252454 14130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:42.260682 14143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:42.260711 14140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:42.260905 14130 server_base.cc:1034] running on GCE node
W20260809 09:27:42.260954 14137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:42.261328 14130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:42.261862 14130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:42.263032 14130 hybrid_clock.cc:648] HybridClock initialized: now 1786267662263008 us; error 36 us; skew 500 ppm
Aug 09 09:27:42 dist-test-slave-7vjv krb5kdc[13707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267662, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:42.265542 14130 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:27:42.266295 14130 webserver.cc:460] Webserver started at http://127.9.102.2:37997/ using document root <none> and password file <none>
I20260809 09:27:42.266748 14130 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:42.266827 14130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:42.267022 14130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:42.269049 14130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "4f47f9e2016e49e99cf22e9986b030b5"
format_stamp: "Formatted at 2026-08-09 09:27:42 on dist-test-slave-7vjv"
server_key: "df80b1f584e9923329f6e7e8709e9cd1"
server_key_iv: "a0657021cb0d35d85219548f5df310b5"
server_key_version: "encryptionkey@0"
I20260809 09:27:42.269593 14130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "4f47f9e2016e49e99cf22e9986b030b5"
format_stamp: "Formatted at 2026-08-09 09:27:42 on dist-test-slave-7vjv"
server_key: "df80b1f584e9923329f6e7e8709e9cd1"
server_key_iv: "a0657021cb0d35d85219548f5df310b5"
server_key_version: "encryptionkey@0"
I20260809 09:27:42.272761 14130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:27:42.274998 14154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:42.275979 14130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:42.276105 14130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "4f47f9e2016e49e99cf22e9986b030b5"
format_stamp: "Formatted at 2026-08-09 09:27:42 on dist-test-slave-7vjv"
server_key: "df80b1f584e9923329f6e7e8709e9cd1"
server_key_iv: "a0657021cb0d35d85219548f5df310b5"
server_key_version: "encryptionkey@0"
I20260809 09:27:42.276248 14130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:42.293942 14130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:42.296456 14130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:42.296633 14130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:42.297109 14130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:42.297796 14130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:42.297868 14130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:42.297926 14130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:42.297966 14130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:42.305888 14130 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:42301
I20260809 09:27:42.305917 14351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:42301 every 8 connection(s)
I20260809 09:27:42.306612 14130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:27:42.312186 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 14130
I20260809 09:27:42.312265 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:27:42.312520 9624 external_mini_cluster.cc:1428] Setting key f5aa9bdfaec3b81903dccdc25ab4b6fb
Aug 09 09:27:42 dist-test-slave-7vjv krb5kdc[13707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267662, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:42.320310 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:42.308275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:35453 (local address 127.9.102.62:42513)
0809 09:27:42.308422 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:42.308426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:42.309185 (+ 759us) server_negotiation.cc:407] Connection header received
0809 09:27:42.310224 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:42.310226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:42.310270 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:42.310356 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:42.311333 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:42.311842 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:42.313128 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:42.313364 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:42.313444 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:42.316313 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:42.316320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:42.316324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:42.316357 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:42.318026 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:42.318528 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:42.318533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:42.318534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:42.318626 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:42.318929 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:42.318935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:42.318939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:42.319241 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:42.319352 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:42.319899 (+ 547us) server_negotiation.cc:299] Negotiation successful
0809 09:27:42.320048 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 09:27:42.320712 14355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:42.308628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:42513 (local address 127.9.102.2:35453)
0809 09:27:42.309052 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:42.309087 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:42.310041 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:42.310502 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:42.310507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:42.310819 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:42.311215 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:42.311227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:42.311985 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:42.311987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:42.313017 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:42.313029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:42.313511 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:42.313515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:42.313687 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:42.314299 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:42.314316 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:42.316106 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:42.318164 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:42.318169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:42.318179 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:42.318418 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:42.318711 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:42.318713 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:42.318714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:42.318831 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:42.319372 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:42.319380 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:42.319644 (+ 264us) client_negotiation.cc:769] Sending connection context
0809 09:27:42.319788 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 09:27:42.320001 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20260809 09:27:42.321736 14352 heartbeater.cc:347] Connected to a master server at 127.9.102.62:42513
I20260809 09:27:42.321933 14352 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:42.322330 14352 heartbeater.cc:511] Master 127.9.102.62:42513 requested a full tablet report, sending...
I20260809 09:27:42.323254 13784 ts_manager.cc:194] Registered new tserver with Master: 4f47f9e2016e49e99cf22e9986b030b5 (127.9.102.2:42301)
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
I20260809 09:27:42.323720 13784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:35453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:27:42.340196 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:42513
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:42.437633 14358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:42.437865 14358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:42.437930 14358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:42.441783 14358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:42.441856 14358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:42.441937 14358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:27:42.446511 14358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:42513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:42.447327 14358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:42.448205 14358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:42.456584 14364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:42.456622 14365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:42.456635 14368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:42.456753 14358 server_base.cc:1034] running on GCE node
I20260809 09:27:42.457064 14358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:42.457690 14358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:42.458878 14358 hybrid_clock.cc:648] HybridClock initialized: now 1786267662458846 us; error 42 us; skew 500 ppm
Aug 09 09:27:42 dist-test-slave-7vjv krb5kdc[13707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267662, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:42.461434 14358 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:27:42.462147 14358 webserver.cc:460] Webserver started at http://127.9.102.3:45925/ using document root <none> and password file <none>
I20260809 09:27:42.462615 14358 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:42.462673 14358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:42.462867 14358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:42.464655 14358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "8b545c9bf70240ee89e80b8029843245"
format_stamp: "Formatted at 2026-08-09 09:27:42 on dist-test-slave-7vjv"
server_key: "ac4db0d79727c0a23dad00327e717ed7"
server_key_iv: "66239d124334adb9a1c2d8e7ada0e8fc"
server_key_version: "encryptionkey@0"
I20260809 09:27:42.465139 14358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "8b545c9bf70240ee89e80b8029843245"
format_stamp: "Formatted at 2026-08-09 09:27:42 on dist-test-slave-7vjv"
server_key: "ac4db0d79727c0a23dad00327e717ed7"
server_key_iv: "66239d124334adb9a1c2d8e7ada0e8fc"
server_key_version: "encryptionkey@0"
I20260809 09:27:42.468156 14358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:27:42.470064 14384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:42.471132 14358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:42.471297 14358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "8b545c9bf70240ee89e80b8029843245"
format_stamp: "Formatted at 2026-08-09 09:27:42 on dist-test-slave-7vjv"
server_key: "ac4db0d79727c0a23dad00327e717ed7"
server_key_iv: "66239d124334adb9a1c2d8e7ada0e8fc"
server_key_version: "encryptionkey@0"
I20260809 09:27:42.471401 14358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:42.508720 14358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:42.511236 14358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:42.511426 14358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:42.511929 14358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:42.512696 14358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:42.512764 14358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:42.512838 14358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:42.512882 14358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:42.520756 14358 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:38477
I20260809 09:27:42.520780 14576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:38477 every 8 connection(s)
I20260809 09:27:42.521543 14358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:27:42.526361 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 14358
I20260809 09:27:42.526461 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:27:42.526719 9624 external_mini_cluster.cc:1428] Setting key 86679afdbd0dea8817872a18545b54fd
Aug 09 09:27:42 dist-test-slave-7vjv krb5kdc[13707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267662, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:42.534201 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:42.523134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:40737 (local address 127.9.102.62:42513)
0809 09:27:42.523285 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:42.523290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:42.523989 (+ 699us) server_negotiation.cc:407] Connection header received
0809 09:27:42.525118 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:42.525121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:42.525164 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:42.525233 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:42.526213 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:42.526932 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:42.528224 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:42.528418 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:42.528510 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:42.530712 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:42.530717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:42.530723 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:42.530744 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:42.531834 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:42.532358 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:42.532362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:42.532364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:42.532442 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:42.532812 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:42.532815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:42.532817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:42.533090 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:42.533206 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:42.533869 (+ 663us) server_negotiation.cc:299] Negotiation successful
0809 09:27:42.534062 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 09:27:42.534746 14581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:42.523456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:42513 (local address 127.9.102.3:40737)
0809 09:27:42.523850 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:42.523886 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:42.524939 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:42.525373 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:42.525378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:42.525663 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:42.526061 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:42.526073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:42.527101 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:42.527104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:42.528079 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:42.528088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:42.528561 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:42.528565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:42.528707 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:42.529220 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:42.529234 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:42.530559 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:42.531941 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:42.531947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:42.531958 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:42.532266 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:42.532565 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:42.532568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:42.532570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:42.532704 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:42.533236 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:42.533242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:42.533546 (+ 304us) client_negotiation.cc:769] Sending connection context
0809 09:27:42.533734 (+ 188us) client_negotiation.cc:240] Negotiation successful
0809 09:27:42.533998 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":106,"threads_started":1}
I20260809 09:27:42.535724 14578 heartbeater.cc:347] Connected to a master server at 127.9.102.62:42513
I20260809 09:27:42.535928 14578 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:42.536373 14578 heartbeater.cc:511] Master 127.9.102.62:42513 requested a full tablet report, sending...
I20260809 09:27:42.537282 13784 ts_manager.cc:194] Registered new tserver with Master: 8b545c9bf70240ee89e80b8029843245 (127.9.102.3:38477)
I20260809 09:27:42.537741 13784 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:40737
I20260809 09:27:42.540169 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 09:27:42.553889 13848 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:27:42.552709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.9.102.62:42513)
0809 09:27:42.552815 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:42.552818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:42.553040 (+ 222us) server_negotiation.cc:407] Connection header received
0809 09:27:42.553145 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:42.553146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:42.553178 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:42.553240 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:42.553793 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39038: BlockingRecv error: recv got EOF from 127.0.0.1:39038 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:27:42.554800 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 13884
I20260809 09:27:42.594285 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 14130
I20260809 09:27:42.607755 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 14358
I20260809 09:27:42.617740 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 13727
2026-08-09T09:27:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2183 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:27:42 dist-test-slave-7vjv krb5kdc[14596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:27:42 dist-test-slave-7vjv krb5kdc[14596](info): set up 2 sockets
Aug 09 09:27:42 dist-test-slave-7vjv krb5kdc[14596](info): commencing operation
krb5kdc: starting...
W20260809 09:27:43.766850 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:27:43 dist-test-slave-7vjv krb5kdc[14596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:27:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:27:43.876515 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:42371
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:42371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:43.975659 14616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:43.975862 14616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:43.975915 14616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:43.979514 14616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:43.979560 14616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:43.979576 14616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:43.979619 14616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:43.979632 14616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:43.984416 14616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:42371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:42371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:43.985301 14616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:43.986171 14616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:43.993706 14621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:43.993710 14623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:43.993865 14616 server_base.cc:1034] running on GCE node
W20260809 09:27:43.993757 14626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:43.994445 14616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:43.995391 14616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:43.996564 14616 hybrid_clock.cc:648] HybridClock initialized: now 1786267663996547 us; error 27 us; skew 500 ppm
Aug 09 09:27:43 dist-test-slave-7vjv krb5kdc[14596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267663, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:44.002645 14616 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:27:44.003422 14616 webserver.cc:460] Webserver started at http://127.9.102.62:42657/ using document root <none> and password file <none>
I20260809 09:27:44.003916 14616 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:44.003968 14616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:44.004174 14616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:44.005993 14616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "b86044b50766484abd96c38cf6b3da01"
format_stamp: "Formatted at 2026-08-09 09:27:44 on dist-test-slave-7vjv"
server_key: "4c7d3cfd1754199529476c7c54af3cde"
server_key_iv: "5b38bcc41051929bcb5f1e2818b043b9"
server_key_version: "encryptionkey@0"
I20260809 09:27:44.006474 14616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "b86044b50766484abd96c38cf6b3da01"
format_stamp: "Formatted at 2026-08-09 09:27:44 on dist-test-slave-7vjv"
server_key: "4c7d3cfd1754199529476c7c54af3cde"
server_key_iv: "5b38bcc41051929bcb5f1e2818b043b9"
server_key_version: "encryptionkey@0"
I20260809 09:27:44.009792 14616 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 09:27:44.011822 14636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:44.012956 14616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 09:27:44.013051 14616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "b86044b50766484abd96c38cf6b3da01"
format_stamp: "Formatted at 2026-08-09 09:27:44 on dist-test-slave-7vjv"
server_key: "4c7d3cfd1754199529476c7c54af3cde"
server_key_iv: "5b38bcc41051929bcb5f1e2818b043b9"
server_key_version: "encryptionkey@0"
I20260809 09:27:44.013128 14616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:44.022838 14616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:44.025286 14616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:44.025409 14616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:44.032366 14616 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:42371
I20260809 09:27:44.032640 14730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:42371 every 8 connection(s)
I20260809 09:27:44.033201 14616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:44.035707 14731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:44.040634 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01: Bootstrap starting.
I20260809 09:27:44.041280 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 14616
I20260809 09:27:44.041402 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:44.041675 9624 external_mini_cluster.cc:1428] Setting key 665716d73d7e33bf036d46567e8516f4
I20260809 09:27:44.043114 14731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:44.043900 14731 log.cc:824] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:44.045578 14731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01: No bootstrap required, opened a new log
I20260809 09:27:44.048623 14731 raft_consensus.cc:348] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b86044b50766484abd96c38cf6b3da01" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42371 } }
I20260809 09:27:44.048823 14731 raft_consensus.cc:374] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:44.048894 14731 raft_consensus.cc:729] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b86044b50766484abd96c38cf6b3da01, State: Initialized, Role: FOLLOWER
I20260809 09:27:44.049353 14731 consensus_queue.cc:260] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b86044b50766484abd96c38cf6b3da01" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42371 } }
I20260809 09:27:44.049515 14731 raft_consensus.cc:388] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:44.049594 14731 raft_consensus.cc:482] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:44.049683 14731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 09:27:44 dist-test-slave-7vjv krb5kdc[14596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:44.050561 14731 raft_consensus.cc:504] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b86044b50766484abd96c38cf6b3da01" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42371 } }
I20260809 09:27:44.050961 14731 leader_election.cc:302] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b86044b50766484abd96c38cf6b3da01; no voters:
I20260809 09:27:44.051314 14731 leader_election.cc:288] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:44.051451 14740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:44.051648 14740 raft_consensus.cc:686] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [term 1 LEADER]: Becoming Leader. State: Replica: b86044b50766484abd96c38cf6b3da01, State: Running, Role: LEADER
I20260809 09:27:44.051939 14740 consensus_queue.cc:237] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b86044b50766484abd96c38cf6b3da01" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42371 } }
I20260809 09:27:44.052493 14731 sys_catalog.cc:564] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:44.053455 14742 sys_catalog.cc:455] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b86044b50766484abd96c38cf6b3da01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b86044b50766484abd96c38cf6b3da01" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42371 } } }
I20260809 09:27:44.053572 14742 sys_catalog.cc:458] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:44.053860 14743 sys_catalog.cc:455] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b86044b50766484abd96c38cf6b3da01. Latest consensus state: current_term: 1 leader_uuid: "b86044b50766484abd96c38cf6b3da01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b86044b50766484abd96c38cf6b3da01" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42371 } } }
I20260809 09:27:44.053931 14743 sys_catalog.cc:458] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:44.054325 14761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:44.055814 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:44.042721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.9.102.62:42371)
0809 09:27:44.043127 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:44.043146 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:44.043186 (+ 40us) server_negotiation.cc:407] Connection header received
0809 09:27:44.043932 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:44.043953 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:44.044342 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:44.044646 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:44.045440 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:44.046392 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:44.047217 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:44.047638 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:44.047801 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:44.050253 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:44.050259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:44.050276 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:44.050313 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:44.052222 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:44.052680 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:44.052685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:44.052691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:44.052827 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:44.053202 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:44.053208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:44.053212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:44.053744 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:44.054529 (+ 785us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:44.054674 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 09:27:44.055054 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":101,"threads_started":1}
I20260809 09:27:44.057112 14761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:44.058255 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:27:44.062301 14761 catalog_manager.cc:1347] Generated new cluster ID: 068ee60114b745358a6d1107cc2a6103
I20260809 09:27:44.062399 14761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:44.071859 14761 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:44.072932 14761 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:44.076728 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:44.072228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.9.102.62:42371)
0809 09:27:44.072334 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:44.072338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:44.072606 (+ 268us) server_negotiation.cc:407] Connection header received
0809 09:27:44.072709 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:44.072712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:44.072936 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:44.073019 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:44.073521 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:44.074007 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:44.074808 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:44.075012 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:44.075115 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:44.075864 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:44.075867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:44.075870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:44.076268 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:44.076292 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:44.076339 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:44.076438 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:44.076557 (+ 119us) server_negotiation.cc:299] Negotiation successful
0809 09:27:44.076603 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 09:27:44.077653 14761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b86044b50766484abd96c38cf6b3da01: Generated new TSK 0
I20260809 09:27:44.078305 14761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:27:44.084476 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 14616
2026-08-09T09:27:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1472 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:27:44 dist-test-slave-7vjv krb5kdc[14791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:27:44 dist-test-slave-7vjv krb5kdc[14791](info): set up 2 sockets
Aug 09 09:27:44 dist-test-slave-7vjv krb5kdc[14791](info): commencing operation
krb5kdc: starting...
W20260809 09:27:45.262132 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:27:45 dist-test-slave-7vjv krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:27:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:27:45.371155 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39257
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:45.470075 14809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:45.470263 14809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:45.470296 14809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:45.473682 14809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:45.473721 14809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:45.473737 14809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:45.473752 14809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:45.473764 14809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:45.478176 14809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:45.478979 14809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:45.479792 14809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:45.487149 14818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:45.487212 14820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:45.487366 14809 server_base.cc:1034] running on GCE node
W20260809 09:27:45.487149 14816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:45.488001 14809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:45.488970 14809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:45.490124 14809 hybrid_clock.cc:648] HybridClock initialized: now 1786267665490108 us; error 27 us; skew 500 ppm
Aug 09 09:27:45 dist-test-slave-7vjv krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267665, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:45.493002 14809 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:27:45.493747 14809 webserver.cc:460] Webserver started at http://127.9.102.62:41549/ using document root <none> and password file <none>
I20260809 09:27:45.494217 14809 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:45.494300 14809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:45.494478 14809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:45.496366 14809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "91eecaf7fab143c9bc3674690273612c"
format_stamp: "Formatted at 2026-08-09 09:27:45 on dist-test-slave-7vjv"
server_key: "bc45d661cdcaa7cf7328510e62742d9d"
server_key_iv: "7eb09d7bbb9a33673dafb88800568027"
server_key_version: "encryptionkey@0"
I20260809 09:27:45.496907 14809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "91eecaf7fab143c9bc3674690273612c"
format_stamp: "Formatted at 2026-08-09 09:27:45 on dist-test-slave-7vjv"
server_key: "bc45d661cdcaa7cf7328510e62742d9d"
server_key_iv: "7eb09d7bbb9a33673dafb88800568027"
server_key_version: "encryptionkey@0"
I20260809 09:27:45.500003 14809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:27:45.502107 14832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:45.503093 14809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:45.503222 14809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "91eecaf7fab143c9bc3674690273612c"
format_stamp: "Formatted at 2026-08-09 09:27:45 on dist-test-slave-7vjv"
server_key: "bc45d661cdcaa7cf7328510e62742d9d"
server_key_iv: "7eb09d7bbb9a33673dafb88800568027"
server_key_version: "encryptionkey@0"
I20260809 09:27:45.503319 14809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:45.512630 14809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:45.515331 14809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:45.515506 14809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:45.521569 14809 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:39257
I20260809 09:27:45.521832 14921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:39257 every 8 connection(s)
I20260809 09:27:45.522416 14809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:45.524768 14923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:45.526554 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 14809
I20260809 09:27:45.526676 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:45.526923 9624 external_mini_cluster.cc:1428] Setting key 966ffc4be7e08de559027b24485e07b7
I20260809 09:27:45.530372 14923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c: Bootstrap starting.
I20260809 09:27:45.532606 14923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:45.533361 14923 log.cc:824] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:45.535166 14923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c: No bootstrap required, opened a new log
Aug 09 09:27:45 dist-test-slave-7vjv krb5kdc[14791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:45.538214 14923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91eecaf7fab143c9bc3674690273612c" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39257 } }
I20260809 09:27:45.538425 14923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:45.538484 14923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91eecaf7fab143c9bc3674690273612c, State: Initialized, Role: FOLLOWER
I20260809 09:27:45.538938 14923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91eecaf7fab143c9bc3674690273612c" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39257 } }
I20260809 09:27:45.539099 14923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:45.539178 14923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:45.539297 14923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:45.540063 14923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91eecaf7fab143c9bc3674690273612c" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39257 } }
I20260809 09:27:45.540438 14923 leader_election.cc:302] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91eecaf7fab143c9bc3674690273612c; no voters:
I20260809 09:27:45.540714 14923 leader_election.cc:288] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:45.540835 14935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:45.541069 14935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [term 1 LEADER]: Becoming Leader. State: Replica: 91eecaf7fab143c9bc3674690273612c, State: Running, Role: LEADER
I20260809 09:27:45.541240 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:45.528162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.9.102.62:39257)
0809 09:27:45.528533 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:45.528550 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:45.528586 (+ 36us) server_negotiation.cc:407] Connection header received
0809 09:27:45.529417 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:45.529439 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:45.529803 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:45.530175 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:45.530850 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:45.532192 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:45.532952 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:45.533573 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:45.533738 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:45.536028 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:45.536033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:45.536044 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:45.536067 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:45.538086 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:45.538577 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:45.538582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:45.538586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:45.538692 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:45.538982 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:45.538986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:45.538988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:45.539478 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:45.539704 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:45.540001 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 09:27:45.540335 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":90,"threads_started":1}
I20260809 09:27:45.541406 14935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91eecaf7fab143c9bc3674690273612c" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39257 } }
I20260809 09:27:45.541882 14923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:45.542495 14940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91eecaf7fab143c9bc3674690273612c. Latest consensus state: current_term: 1 leader_uuid: "91eecaf7fab143c9bc3674690273612c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91eecaf7fab143c9bc3674690273612c" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39257 } } }
I20260809 09:27:45.542677 14940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:45.543274 14949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:45.544595 14937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91eecaf7fab143c9bc3674690273612c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91eecaf7fab143c9bc3674690273612c" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39257 } } }
I20260809 09:27:45.544690 14937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:45.546291 14949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:45.551620 14949 catalog_manager.cc:1347] Generated new cluster ID: ce64cad504254d1bae5be00e24adf0d3
I20260809 09:27:45.551690 14949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:45.589960 14949 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:45.590826 14949 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:45.598994 14949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91eecaf7fab143c9bc3674690273612c: Generated new TSK 0
I20260809 09:27:45.599573 14949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:27:45.636600 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39257
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:45.736811 14966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:45.737028 14966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:45.737067 14966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:45.740435 14966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:45.740478 14966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:45.740526 14966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:27:45.745462 14966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:45.746204 14966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:45.746961 14966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:45.755352 14979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:45.755352 14976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:45.755414 14973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:45.755550 14966 server_base.cc:1034] running on GCE node
I20260809 09:27:45.756059 14966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:45.756628 14966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:45.757781 14966 hybrid_clock.cc:648] HybridClock initialized: now 1786267665757771 us; error 34 us; skew 500 ppm
Aug 09 09:27:45 dist-test-slave-7vjv krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267665, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:45.760210 14966 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:27:45.760953 14966 webserver.cc:460] Webserver started at http://127.9.102.1:40445/ using document root <none> and password file <none>
I20260809 09:27:45.761423 14966 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:45.761478 14966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:45.761691 14966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:45.763470 14966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "786ad1f016894f668d95393df33cc83f"
format_stamp: "Formatted at 2026-08-09 09:27:45 on dist-test-slave-7vjv"
server_key: "bdd73437f167cb541fa0a1f80738e073"
server_key_iv: "0a6b7b86fff49abbe173cfa3263234e2"
server_key_version: "encryptionkey@0"
I20260809 09:27:45.763988 14966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "786ad1f016894f668d95393df33cc83f"
format_stamp: "Formatted at 2026-08-09 09:27:45 on dist-test-slave-7vjv"
server_key: "bdd73437f167cb541fa0a1f80738e073"
server_key_iv: "0a6b7b86fff49abbe173cfa3263234e2"
server_key_version: "encryptionkey@0"
I20260809 09:27:45.767361 14966 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 09:27:45.769289 14990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:45.770257 14966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:45.770347 14966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "786ad1f016894f668d95393df33cc83f"
format_stamp: "Formatted at 2026-08-09 09:27:45 on dist-test-slave-7vjv"
server_key: "bdd73437f167cb541fa0a1f80738e073"
server_key_iv: "0a6b7b86fff49abbe173cfa3263234e2"
server_key_version: "encryptionkey@0"
I20260809 09:27:45.770416 14966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:45.786257 14966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:45.788784 14966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:45.788910 14966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:45.789350 14966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:45.790006 14966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:45.790060 14966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:45.790094 14966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:45.790109 14966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:45.798038 14966 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:33585
I20260809 09:27:45.798087 15198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:33585 every 8 connection(s)
I20260809 09:27:45.798745 14966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:27:45.801957 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 14966
I20260809 09:27:45.802058 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:27:45.802335 9624 external_mini_cluster.cc:1428] Setting key 97fd1e1ddb4de17e358a8bd22d12ca59
Aug 09 09:27:45 dist-test-slave-7vjv krb5kdc[14791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267665, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:45.811964 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:45.800261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:60133 (local address 127.9.102.62:39257)
0809 09:27:45.800392 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:45.800395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:45.801055 (+ 660us) server_negotiation.cc:407] Connection header received
0809 09:27:45.802466 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:45.802469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:45.802521 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:45.802585 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:45.803852 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:45.804366 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:45.805953 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:45.806301 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:45.806404 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:45.808694 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:45.808697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:45.808699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:45.808720 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:45.810098 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:45.810625 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:45.810629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:45.810631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:45.810706 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:45.811022 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:45.811025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:45.811027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:45.811186 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:45.811284 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:45.811702 (+ 418us) server_negotiation.cc:299] Negotiation successful
0809 09:27:45.811820 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:27:45.812666 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:45.800526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39257 (local address 127.9.102.1:60133)
0809 09:27:45.800916 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:45.800952 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:45.802211 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:45.802781 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:45.802787 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:45.803186 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:45.803697 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:45.803727 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:45.804515 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:45.804521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:45.805818 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:45.805833 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:45.806450 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:45.806453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:45.806579 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:45.807084 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:45.807098 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:45.808549 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:45.810231 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:45.810237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:45.810248 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:45.810482 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:45.810792 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:45.810795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:45.810797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:45.810923 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:45.811286 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:45.811292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:45.811496 (+ 204us) client_negotiation.cc:769] Sending connection context
0809 09:27:45.811662 (+ 166us) client_negotiation.cc:240] Negotiation successful
0809 09:27:45.811922 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260809 09:27:45.813700 15200 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39257
I20260809 09:27:45.813897 15200 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:45.814314 15200 heartbeater.cc:511] Master 127.9.102.62:39257 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
I20260809 09:27:45.815833 14857 ts_manager.cc:194] Registered new tserver with Master: 786ad1f016894f668d95393df33cc83f (127.9.102.1:33585)
I20260809 09:27:45.817083 14857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:60133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:27:45.828668 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39257
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:45.931391 15206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:45.931622 15206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:45.931659 15206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:45.935184 15206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:45.935230 15206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:45.935281 15206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:27:45.939599 15206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:45.940375 15206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:45.941160 15206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:45.949627 15218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:45.949617 15222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:45.949620 15219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:45.950309 15206 server_base.cc:1034] running on GCE node
I20260809 09:27:45.950686 15206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:45.951257 15206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:45.952409 15206 hybrid_clock.cc:648] HybridClock initialized: now 1786267665952393 us; error 33 us; skew 500 ppm
Aug 09 09:27:45 dist-test-slave-7vjv krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267665, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:45.954850 15206 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:27:45.955627 15206 webserver.cc:460] Webserver started at http://127.9.102.2:36087/ using document root <none> and password file <none>
I20260809 09:27:45.956216 15206 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:45.956301 15206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:45.956494 15206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:45.958227 15206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "c40aa454d4054a6985d46ee81d1d0868"
format_stamp: "Formatted at 2026-08-09 09:27:45 on dist-test-slave-7vjv"
server_key: "5e155ce43d714d01831dc9b4ef5aeed8"
server_key_iv: "24321c3230cd87c7e3c23b58a180b41b"
server_key_version: "encryptionkey@0"
I20260809 09:27:45.958748 15206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "c40aa454d4054a6985d46ee81d1d0868"
format_stamp: "Formatted at 2026-08-09 09:27:45 on dist-test-slave-7vjv"
server_key: "5e155ce43d714d01831dc9b4ef5aeed8"
server_key_iv: "24321c3230cd87c7e3c23b58a180b41b"
server_key_version: "encryptionkey@0"
I20260809 09:27:45.962023 15206 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:45.963984 15232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:45.965000 15206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:45.965142 15206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "c40aa454d4054a6985d46ee81d1d0868"
format_stamp: "Formatted at 2026-08-09 09:27:45 on dist-test-slave-7vjv"
server_key: "5e155ce43d714d01831dc9b4ef5aeed8"
server_key_iv: "24321c3230cd87c7e3c23b58a180b41b"
server_key_version: "encryptionkey@0"
I20260809 09:27:45.965235 15206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:45.986332 15206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:45.989265 15206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:45.989434 15206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:45.989890 15206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:45.990581 15206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:45.990648 15206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:45.990710 15206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:45.990777 15206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:45.998314 15206 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:40599
I20260809 09:27:45.998335 15427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:40599 every 8 connection(s)
I20260809 09:27:45.999025 15206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:27:46.003957 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 15206
I20260809 09:27:46.004109 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:27:46.004361 9624 external_mini_cluster.cc:1428] Setting key 743f76ce175b672ba937e39ec570c4f2
Aug 09 09:27:46 dist-test-slave-7vjv krb5kdc[14791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267665, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:46.011891 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:46.000518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:51737 (local address 127.9.102.62:39257)
0809 09:27:46.000649 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:46.000652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:46.001300 (+ 648us) server_negotiation.cc:407] Connection header received
0809 09:27:46.002504 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:46.002507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:46.002565 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:46.002674 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:46.003753 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:46.004345 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:46.005689 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:46.005988 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:46.006096 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:46.008560 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:46.008563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:46.008565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:46.008585 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:46.009971 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:46.010555 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:46.010558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:46.010559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:46.010643 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:46.010949 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:46.010953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:46.010955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:46.011118 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:46.011214 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:46.011639 (+ 425us) server_negotiation.cc:299] Negotiation successful
0809 09:27:46.011747 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 09:27:46.012579 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:46.000789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39257 (local address 127.9.102.2:51737)
0809 09:27:46.001168 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:46.001202 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:46.002284 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:46.002837 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:46.002843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:46.003156 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:46.003580 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:46.003592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:46.004457 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:46.004466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:46.005573 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:46.005587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:46.006114 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:46.006117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:46.006236 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:46.006795 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:46.006810 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:46.008399 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:46.010110 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:46.010115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:46.010126 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:46.010428 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:46.010726 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:46.010728 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:46.010730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:46.010862 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:46.011214 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:46.011220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:46.011425 (+ 205us) client_negotiation.cc:769] Sending connection context
0809 09:27:46.011586 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 09:27:46.011844 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
I20260809 09:27:46.013614 15428 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39257
I20260809 09:27:46.013836 15428 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
I20260809 09:27:46.014277 15428 heartbeater.cc:511] Master 127.9.102.62:39257 requested a full tablet report, sending...
I20260809 09:27:46.015273 14857 ts_manager.cc:194] Registered new tserver with Master: c40aa454d4054a6985d46ee81d1d0868 (127.9.102.2:40599)
I20260809 09:27:46.015807 14857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:51737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:27:46.030658 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39257
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:46.132817 15438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:46.133009 15438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:46.133052 15438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:46.136511 15438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:46.136564 15438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:46.136610 15438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:27:46.141098 15438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:46.141817 15438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:46.142649 15438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:46.151255 15449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:46.151259 15452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:46.151259 15450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:46.151435 15438 server_base.cc:1034] running on GCE node
I20260809 09:27:46.151919 15438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:46.152511 15438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:46.153684 15438 hybrid_clock.cc:648] HybridClock initialized: now 1786267666153668 us; error 27 us; skew 500 ppm
Aug 09 09:27:46 dist-test-slave-7vjv krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:46.156283 15438 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:27:46.157011 15438 webserver.cc:460] Webserver started at http://127.9.102.3:34831/ using document root <none> and password file <none>
I20260809 09:27:46.157496 15438 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:46.157584 15438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:46.157795 15438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:46.159413 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "f99d161d0c7d4518929ca45aeea0ce2d"
format_stamp: "Formatted at 2026-08-09 09:27:46 on dist-test-slave-7vjv"
server_key: "815225e45d47ec31c38a089bcc55117a"
server_key_iv: "8ae8b41c83c6710eafaadd8128d11bb0"
server_key_version: "encryptionkey@0"
I20260809 09:27:46.159924 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "f99d161d0c7d4518929ca45aeea0ce2d"
format_stamp: "Formatted at 2026-08-09 09:27:46 on dist-test-slave-7vjv"
server_key: "815225e45d47ec31c38a089bcc55117a"
server_key_iv: "8ae8b41c83c6710eafaadd8128d11bb0"
server_key_version: "encryptionkey@0"
I20260809 09:27:46.163058 15438 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 09:27:46.165136 15462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:46.166145 15438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:46.166271 15438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "f99d161d0c7d4518929ca45aeea0ce2d"
format_stamp: "Formatted at 2026-08-09 09:27:46 on dist-test-slave-7vjv"
server_key: "815225e45d47ec31c38a089bcc55117a"
server_key_iv: "8ae8b41c83c6710eafaadd8128d11bb0"
server_key_version: "encryptionkey@0"
I20260809 09:27:46.166363 15438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:46.185032 15438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:46.187297 15438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:46.187474 15438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:46.188045 15438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:46.188835 15438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:46.188915 15438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:46.189002 15438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:46.189059 15438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:46.197043 15438 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:33897
I20260809 09:27:46.197062 15646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:33897 every 8 connection(s)
I20260809 09:27:46.197798 15438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
Aug 09 09:27:46 dist-test-slave-7vjv krb5kdc[14791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267666, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:46.206262 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 15438
I20260809 09:27:46.206369 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:27:46.206701 9624 external_mini_cluster.cc:1428] Setting key ab780fce776dc61be9a022b1e67f3b50
I20260809 09:27:46.210137 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:46.199360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:36937 (local address 127.9.102.62:39257)
0809 09:27:46.199492 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:46.199495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:46.200393 (+ 898us) server_negotiation.cc:407] Connection header received
0809 09:27:46.201236 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:46.201239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:46.201296 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:46.201396 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:46.202400 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:46.202881 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:46.203952 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:46.204240 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:46.204348 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:46.206644 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:46.206648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:46.206650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:46.206675 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:46.208107 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:46.208612 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:46.208618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:46.208621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:46.208718 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:46.209084 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:46.209088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:46.209090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:46.209265 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:46.209373 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:46.209838 (+ 465us) server_negotiation.cc:299] Negotiation successful
0809 09:27:46.209978 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 09:27:46.210812 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:46.199627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39257 (local address 127.9.102.3:36937)
0809 09:27:46.200257 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:46.200294 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:46.201041 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:46.201550 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:46.201557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:46.201868 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:46.202261 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:46.202273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:46.203000 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:46.203003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:46.203824 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:46.203834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:46.204384 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:46.204388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:46.204547 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:46.205059 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:46.205074 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:46.206477 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:46.208251 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:46.208256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:46.208266 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:46.208511 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:46.208819 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:46.208823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:46.208825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:46.208966 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:46.209374 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:46.209380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:46.209613 (+ 233us) client_negotiation.cc:769] Sending connection context
0809 09:27:46.209799 (+ 186us) client_negotiation.cc:240] Negotiation successful
0809 09:27:46.210107 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":93,"threads_started":1}
I20260809 09:27:46.211781 15647 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39257
I20260809 09:27:46.211975 15647 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:46.212404 15647 heartbeater.cc:511] Master 127.9.102.62:39257 requested a full tablet report, sending...
I20260809 09:27:46.213287 14857 ts_manager.cc:194] Registered new tserver with Master: f99d161d0c7d4518929ca45aeea0ce2d (127.9.102.3:33897)
I20260809 09:27:46.213789 14857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:36937
I20260809 09:27:46.220350 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 09:27:46 dist-test-slave-7vjv krb5kdc[14791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267666, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 09:27:46 dist-test-slave-7vjv krb5kdc[14791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267666, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:46.245924 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:46.237749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52142 (local address 127.9.102.62:39257)
0809 09:27:46.237909 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:46.237913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:46.238082 (+ 169us) server_negotiation.cc:407] Connection header received
0809 09:27:46.238186 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:46.238188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:46.238224 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:46.238288 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:46.238918 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:46.239378 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:46.240225 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:46.240418 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:46.240492 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:46.242676 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:46.242679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:46.242681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:46.242700 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:46.244263 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:46.244686 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:46.244690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:46.244691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:46.244781 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:46.245034 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:46.245037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:46.245041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:46.245257 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:46.245394 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:46.245669 (+ 275us) server_negotiation.cc:299] Negotiation successful
0809 09:27:46.245777 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260809 09:27:46.246248 14857 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52142
I20260809 09:27:46.247038 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 14966
I20260809 09:27:46.269920 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 15206
I20260809 09:27:46.280200 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 15438
I20260809 09:27:46.292227 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 14809
2026-08-09T09:27:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2203 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:27:46 dist-test-slave-7vjv krb5kdc[15669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:27:46 dist-test-slave-7vjv krb5kdc[15669](info): set up 2 sockets
Aug 09 09:27:46 dist-test-slave-7vjv krb5kdc[15669](info): commencing operation
krb5kdc: starting...
W20260809 09:27:47.441056 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:27:47 dist-test-slave-7vjv krb5kdc[15669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:27:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:27:47.548854 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:36343
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:36343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:47.648247 15697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:47.648458 15697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:47.648489 15697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:47.651875 15697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:47.651930 15697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:47.651947 15697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:47.651978 15697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:47.651990 15697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:47.656704 15697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:36343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:36343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:47.657512 15697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:47.658377 15697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:47.665769 15713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:47.665791 15708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:47.665810 15707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:47.666030 15697 server_base.cc:1034] running on GCE node
I20260809 09:27:47.666646 15697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:47.667490 15697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:47.668640 15697 hybrid_clock.cc:648] HybridClock initialized: now 1786267667668623 us; error 27 us; skew 500 ppm
Aug 09 09:27:47 dist-test-slave-7vjv krb5kdc[15669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267667, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:47.671068 15697 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:27:47.671782 15697 webserver.cc:460] Webserver started at http://127.9.102.62:35359/ using document root <none> and password file <none>
I20260809 09:27:47.672317 15697 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:47.672411 15697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:47.672611 15697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:47.674335 15697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "69eb7efa2d764841ba7a944cff65d370"
format_stamp: "Formatted at 2026-08-09 09:27:47 on dist-test-slave-7vjv"
server_key: "e0e1d1939155fe238ea563cefb93a82c"
server_key_iv: "5aa62ca8229af856e015ad36f433cc5c"
server_key_version: "encryptionkey@0"
I20260809 09:27:47.674854 15697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "69eb7efa2d764841ba7a944cff65d370"
format_stamp: "Formatted at 2026-08-09 09:27:47 on dist-test-slave-7vjv"
server_key: "e0e1d1939155fe238ea563cefb93a82c"
server_key_iv: "5aa62ca8229af856e015ad36f433cc5c"
server_key_version: "encryptionkey@0"
I20260809 09:27:47.678201 15697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:47.680251 15724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:47.681353 15697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:47.681488 15697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "69eb7efa2d764841ba7a944cff65d370"
format_stamp: "Formatted at 2026-08-09 09:27:47 on dist-test-slave-7vjv"
server_key: "e0e1d1939155fe238ea563cefb93a82c"
server_key_iv: "5aa62ca8229af856e015ad36f433cc5c"
server_key_version: "encryptionkey@0"
I20260809 09:27:47.681591 15697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:47.699616 15697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:47.702158 15697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:47.702329 15697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:47.708459 15697 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:36343
I20260809 09:27:47.708478 15796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:36343 every 8 connection(s)
I20260809 09:27:47.709266 15697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:47.711704 15798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:47.713770 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 15697
I20260809 09:27:47.713866 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:47.714130 9624 external_mini_cluster.cc:1428] Setting key cacbfbb9bb7fd409a48f49e4d1b98206
I20260809 09:27:47.717250 15798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370: Bootstrap starting.
I20260809 09:27:47.719395 15798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:47.720145 15798 log.cc:824] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370: Log is configured to *not* fsync() on all Append() calls
Aug 09 09:27:47 dist-test-slave-7vjv krb5kdc[15669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:47.722034 15798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370: No bootstrap required, opened a new log
I20260809 09:27:47.725210 15798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69eb7efa2d764841ba7a944cff65d370" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 36343 } }
I20260809 09:27:47.725417 15798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:47.725479 15798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69eb7efa2d764841ba7a944cff65d370, State: Initialized, Role: FOLLOWER
I20260809 09:27:47.725936 15798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69eb7efa2d764841ba7a944cff65d370" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 36343 } }
I20260809 09:27:47.726092 15798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:47.726176 15798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:47.726284 15798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:47.726810 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:47.715301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.9.102.62:36343)
0809 09:27:47.715637 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:47.715656 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:47.715686 (+ 30us) server_negotiation.cc:407] Connection header received
0809 09:27:47.716432 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:47.716450 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:47.716727 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:47.716955 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:47.717795 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:47.718592 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:47.719597 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:47.719977 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:47.720087 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:47.722455 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:47.722460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:47.722470 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:47.722492 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:47.723882 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:47.724377 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:47.724382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:47.724385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:47.724482 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:47.724758 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:47.724761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:47.724762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:47.725089 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:47.725283 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:47.725588 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 09:27:47.725846 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":115,"threads_started":1}
I20260809 09:27:47.727258 15798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69eb7efa2d764841ba7a944cff65d370" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 36343 } }
I20260809 09:27:47.727576 15798 leader_election.cc:302] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69eb7efa2d764841ba7a944cff65d370; no voters:
I20260809 09:27:47.727859 15798 leader_election.cc:288] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:47.727965 15805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:47.728259 15805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [term 1 LEADER]: Becoming Leader. State: Replica: 69eb7efa2d764841ba7a944cff65d370, State: Running, Role: LEADER
I20260809 09:27:47.728592 15805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69eb7efa2d764841ba7a944cff65d370" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 36343 } }
I20260809 09:27:47.728950 15798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:47.729648 15805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69eb7efa2d764841ba7a944cff65d370. Latest consensus state: current_term: 1 leader_uuid: "69eb7efa2d764841ba7a944cff65d370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69eb7efa2d764841ba7a944cff65d370" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 36343 } } }
I20260809 09:27:47.729764 15805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:47.730026 15807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69eb7efa2d764841ba7a944cff65d370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69eb7efa2d764841ba7a944cff65d370" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 36343 } } }
I20260809 09:27:47.730137 15807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370 [sys.catalog]: This master's current role is: LEADER
W20260809 09:27:47.733155 15831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:27:47.733217 15831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:27:47.733294 15819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:47.734117 15819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:47.739647 15819 catalog_manager.cc:1347] Generated new cluster ID: 2e53310fc4a2480eaeb11003c3bbc901
I20260809 09:27:47.739706 15819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:47.748194 15819 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:47.749321 15819 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:47.758324 15819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69eb7efa2d764841ba7a944cff65d370: Generated new TSK 0
I20260809 09:27:47.758934 15819 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:27:47.790663 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:36343
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:47.891593 15838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:47.891798 15838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:47.891855 15838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:47.895681 15838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:47.895725 15838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:47.895776 15838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:27:47.900341 15838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:47.901115 15838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:47.901908 15838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:47.910406 15845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:47.910405 15848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:47.910439 15838 server_base.cc:1034] running on GCE node
W20260809 09:27:47.910431 15850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:47.911115 15838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:47.911634 15838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:47.912782 15838 hybrid_clock.cc:648] HybridClock initialized: now 1786267667912753 us; error 38 us; skew 500 ppm
Aug 09 09:27:47 dist-test-slave-7vjv krb5kdc[15669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267667, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:47.915208 15838 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:27:47.915982 15838 webserver.cc:460] Webserver started at http://127.9.102.1:38019/ using document root <none> and password file <none>
I20260809 09:27:47.916508 15838 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:47.916597 15838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:47.916805 15838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:47.918577 15838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "83a0f186ffc5497eb9f8a926b5f5df65"
format_stamp: "Formatted at 2026-08-09 09:27:47 on dist-test-slave-7vjv"
server_key: "858a2b136beee1fda40352a36929a5e3"
server_key_iv: "7dcb791c3e44835f54b8273fbd0b7c19"
server_key_version: "encryptionkey@0"
I20260809 09:27:47.919091 15838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "83a0f186ffc5497eb9f8a926b5f5df65"
format_stamp: "Formatted at 2026-08-09 09:27:47 on dist-test-slave-7vjv"
server_key: "858a2b136beee1fda40352a36929a5e3"
server_key_iv: "7dcb791c3e44835f54b8273fbd0b7c19"
server_key_version: "encryptionkey@0"
I20260809 09:27:47.922478 15838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 09:27:47.924609 15863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:47.925657 15838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:47.925786 15838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "83a0f186ffc5497eb9f8a926b5f5df65"
format_stamp: "Formatted at 2026-08-09 09:27:47 on dist-test-slave-7vjv"
server_key: "858a2b136beee1fda40352a36929a5e3"
server_key_iv: "7dcb791c3e44835f54b8273fbd0b7c19"
server_key_version: "encryptionkey@0"
I20260809 09:27:47.925880 15838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:47.935896 15838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:47.938755 15838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:47.938939 15838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:47.939452 15838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:47.940229 15838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:47.940297 15838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:47.940362 15838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:47.940402 15838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:47.948768 15838 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:33817
I20260809 09:27:47.948796 16062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:33817 every 8 connection(s)
I20260809 09:27:47.949548 15838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:27:47.956025 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 15838
I20260809 09:27:47.956202 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:27:47.956475 9624 external_mini_cluster.cc:1428] Setting key afa0013941c4cbd78e29788943038fc9
Aug 09 09:27:47 dist-test-slave-7vjv krb5kdc[15669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267667, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:47.961951 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:47.951148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:48137 (local address 127.9.102.62:36343)
0809 09:27:47.951276 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:47.951279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:47.951985 (+ 706us) server_negotiation.cc:407] Connection header received
0809 09:27:47.953246 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:47.953249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:47.953303 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:47.953399 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:47.954354 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:47.954847 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:47.955862 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:47.956139 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:47.956255 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:47.958616 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:47.958620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:47.958622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:47.958646 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:47.959942 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:47.960452 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:47.960456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:47.960458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:47.960544 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:47.960918 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:47.960923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:47.960924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:47.961121 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:47.961227 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:47.961663 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 09:27:47.961786 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 09:27:47.962565 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:47.951440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:36343 (local address 127.9.102.1:48137)
0809 09:27:47.951847 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:47.951883 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:47.953054 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:47.953557 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:47.953563 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:47.953844 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:47.954224 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:47.954235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:47.954969 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:47.954972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:47.955727 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:47.955736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:47.956285 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:47.956288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:47.956439 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:47.956992 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:47.957012 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:47.958470 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:47.960056 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:47.960063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:47.960075 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:47.960348 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:47.960682 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:47.960685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:47.960687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:47.960820 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:47.961228 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:47.961235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:47.961440 (+ 205us) client_negotiation.cc:769] Sending connection context
0809 09:27:47.961621 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 09:27:47.961902 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260809 09:27:47.963728 16063 heartbeater.cc:347] Connected to a master server at 127.9.102.62:36343
I20260809 09:27:47.963925 16063 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:47.964362 16063 heartbeater.cc:511] Master 127.9.102.62:36343 requested a full tablet report, sending...
I20260809 09:27:47.965754 15746 ts_manager.cc:194] Registered new tserver with Master: 83a0f186ffc5497eb9f8a926b5f5df65 (127.9.102.1:33817)
I20260809 09:27:47.966784 15746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:48137
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:27:47.982728 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:36343
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:48.082091 16072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:48.082274 16072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:48.082307 16072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:48.085919 16072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:48.085963 16072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:48.086041 16072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:27:48.090349 16072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:48.091185 16072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:48.092051 16072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:48.100278 16086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:48.100276 16090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:48.100350 16072 server_base.cc:1034] running on GCE node
W20260809 09:27:48.100276 16088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:48.100904 16072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:48.101454 16072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:48.102625 16072 hybrid_clock.cc:648] HybridClock initialized: now 1786267668102609 us; error 28 us; skew 500 ppm
Aug 09 09:27:48 dist-test-slave-7vjv krb5kdc[15669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267668, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:48.105150 16072 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:27:48.105907 16072 webserver.cc:460] Webserver started at http://127.9.102.2:41743/ using document root <none> and password file <none>
I20260809 09:27:48.106346 16072 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:48.106401 16072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:48.106550 16072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:48.108383 16072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "4b81197c023740f58632e88e9c06fe25"
format_stamp: "Formatted at 2026-08-09 09:27:48 on dist-test-slave-7vjv"
server_key: "8d685b6dd931d737ec8042a2ac5e20f7"
server_key_iv: "c6e6a35705c5f5df31af57947dc22e69"
server_key_version: "encryptionkey@0"
I20260809 09:27:48.108832 16072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "4b81197c023740f58632e88e9c06fe25"
format_stamp: "Formatted at 2026-08-09 09:27:48 on dist-test-slave-7vjv"
server_key: "8d685b6dd931d737ec8042a2ac5e20f7"
server_key_iv: "c6e6a35705c5f5df31af57947dc22e69"
server_key_version: "encryptionkey@0"
I20260809 09:27:48.112088 16072 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 09:27:48.114158 16102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:48.115151 16072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 09:27:48.115271 16072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "4b81197c023740f58632e88e9c06fe25"
format_stamp: "Formatted at 2026-08-09 09:27:48 on dist-test-slave-7vjv"
server_key: "8d685b6dd931d737ec8042a2ac5e20f7"
server_key_iv: "c6e6a35705c5f5df31af57947dc22e69"
server_key_version: "encryptionkey@0"
I20260809 09:27:48.115378 16072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:48.124043 16072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:48.126293 16072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:48.126456 16072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:48.126909 16072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:48.127557 16072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:48.127619 16072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:48.127676 16072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:48.127717 16072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:48.135771 16072 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:34745
I20260809 09:27:48.135794 16290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:34745 every 8 connection(s)
I20260809 09:27:48.136520 16072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:27:48.138437 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 16072
I20260809 09:27:48.138536 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:27:48.138863 9624 external_mini_cluster.cc:1428] Setting key a7427147f31bfd1dc6aa688886740add
Aug 09 09:27:48 dist-test-slave-7vjv krb5kdc[15669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267668, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
I20260809 09:27:48.150476 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:48.138033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:54873 (local address 127.9.102.62:36343)
0809 09:27:48.138180 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:48.138184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:48.139084 (+ 900us) server_negotiation.cc:407] Connection header received
0809 09:27:48.140730 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:48.140733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:48.140772 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:48.140832 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:48.142281 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:48.142819 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:48.144412 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:48.144626 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:48.144702 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:48.147269 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:48.147274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:48.147281 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:48.147301 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:48.148628 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:48.149116 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:48.149121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:48.149123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:48.149202 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:48.149531 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:48.149534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:48.149536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:48.149696 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:48.149795 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:48.150210 (+ 415us) server_negotiation.cc:299] Negotiation successful
0809 09:27:48.150328 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 09:27:48.151144 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:48.138382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:36343 (local address 127.9.102.2:54873)
0809 09:27:48.138905 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:48.138957 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:48.140508 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:48.141012 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:48.141023 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:48.141516 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:48.142137 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:48.142158 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:48.142941 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:48.142946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:48.144292 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:48.144306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:48.144778 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:48.144785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:48.145001 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:48.145536 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:48.145562 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:48.147123 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:48.148738 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:48.148743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:48.148772 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:48.149012 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:48.149294 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:48.149298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:48.149300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:48.149427 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:48.149795 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:48.149802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:48.150002 (+ 200us) client_negotiation.cc:769] Sending connection context
0809 09:27:48.150167 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 09:27:48.150427 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":174,"threads_started":1}
I20260809 09:27:48.152154 16291 heartbeater.cc:347] Connected to a master server at 127.9.102.62:36343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
I20260809 09:27:48.152397 16291 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:48.152926 16291 heartbeater.cc:511] Master 127.9.102.62:36343 requested a full tablet report, sending...
I20260809 09:27:48.154065 15746 ts_manager.cc:194] Registered new tserver with Master: 4b81197c023740f58632e88e9c06fe25 (127.9.102.2:34745)
I20260809 09:27:48.154588 15746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:54873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:27:48.166237 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:36343
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:39655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:48.267567 16300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:48.267771 16300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:48.267832 16300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:48.271466 16300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:48.271517 16300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:48.271618 16300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:27:48.276208 16300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:39655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:48.277035 16300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:48.277927 16300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:48.286289 16308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:48.286294 16315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:48.286303 16311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:48.286463 16300 server_base.cc:1034] running on GCE node
I20260809 09:27:48.286840 16300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:48.287366 16300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:48.288543 16300 hybrid_clock.cc:648] HybridClock initialized: now 1786267668288521 us; error 37 us; skew 500 ppm
Aug 09 09:27:48 dist-test-slave-7vjv krb5kdc[15669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267668, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:48.291113 16300 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:27:48.291841 16300 webserver.cc:460] Webserver started at http://127.9.102.3:43847/ using document root <none> and password file <none>
I20260809 09:27:48.292421 16300 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:48.292508 16300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:48.292771 16300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:48.294548 16300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "da1641e9b8a34d0892df590ba104c600"
format_stamp: "Formatted at 2026-08-09 09:27:48 on dist-test-slave-7vjv"
server_key: "26f58184d362bb40eda3648a5b8ec00c"
server_key_iv: "d818cd872ac20daeb974ae63e9a821d6"
server_key_version: "encryptionkey@0"
I20260809 09:27:48.295090 16300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "da1641e9b8a34d0892df590ba104c600"
format_stamp: "Formatted at 2026-08-09 09:27:48 on dist-test-slave-7vjv"
server_key: "26f58184d362bb40eda3648a5b8ec00c"
server_key_iv: "d818cd872ac20daeb974ae63e9a821d6"
server_key_version: "encryptionkey@0"
I20260809 09:27:48.298430 16300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260809 09:27:48.300601 16330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:48.301682 16300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260809 09:27:48.301808 16300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "da1641e9b8a34d0892df590ba104c600"
format_stamp: "Formatted at 2026-08-09 09:27:48 on dist-test-slave-7vjv"
server_key: "26f58184d362bb40eda3648a5b8ec00c"
server_key_iv: "d818cd872ac20daeb974ae63e9a821d6"
server_key_version: "encryptionkey@0"
I20260809 09:27:48.301906 16300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:48.326959 16300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:48.329517 16300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:48.329756 16300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:48.330232 16300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:48.331014 16300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:48.331086 16300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:48.331146 16300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:48.331180 16300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:48.340888 16300 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:39953
I20260809 09:27:48.341033 16519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:39953 every 8 connection(s)
I20260809 09:27:48.341853 16300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
Aug 09 09:27:48 dist-test-slave-7vjv krb5kdc[15669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267668, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:48.351634 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 16300
I20260809 09:27:48.351751 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:27:48.352073 9624 external_mini_cluster.cc:1428] Setting key 0cdfabaef948916ac7894ea071a4ea26
I20260809 09:27:48.354044 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:48.343380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:46993 (local address 127.9.102.62:36343)
0809 09:27:48.343500 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:48.343504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:48.344149 (+ 645us) server_negotiation.cc:407] Connection header received
0809 09:27:48.345344 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:48.345347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:48.345400 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:48.345466 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:48.346498 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:48.347033 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:48.348132 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:48.348348 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:48.348446 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:48.350393 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:48.350396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:48.350398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:48.350419 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:48.351881 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:48.352480 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:48.352484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:48.352485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:48.352582 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:48.352974 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:48.352977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:48.352978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:48.353165 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:48.353288 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:48.353788 (+ 500us) server_negotiation.cc:299] Negotiation successful
0809 09:27:48.353894 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 09:27:48.354663 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:48.343623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:36343 (local address 127.9.102.3:46993)
0809 09:27:48.343993 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:48.344027 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:48.345156 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:48.345606 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:48.345612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:48.345916 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:48.346337 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:48.346348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:48.347161 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:48.347164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:48.347986 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:48.347995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:48.348478 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:48.348482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:48.348627 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:48.349025 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:48.349044 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:48.350255 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:48.351989 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:48.351996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:48.352021 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:48.352376 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:48.352703 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:48.352706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:48.352708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:48.352858 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:48.353295 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:48.353299 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:48.353574 (+ 275us) client_negotiation.cc:769] Sending connection context
0809 09:27:48.353747 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 09:27:48.353999 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":103,"threads_started":1}
I20260809 09:27:48.355573 16520 heartbeater.cc:347] Connected to a master server at 127.9.102.62:36343
I20260809 09:27:48.355863 16520 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:48.356344 16520 heartbeater.cc:511] Master 127.9.102.62:36343 requested a full tablet report, sending...
I20260809 09:27:48.357260 15746 ts_manager.cc:194] Registered new tserver with Master: da1641e9b8a34d0892df590ba104c600 (127.9.102.3:39953)
I20260809 09:27:48.357780 15746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:46993
I20260809 09:27:48.365541 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 09:27:48 dist-test-slave-7vjv krb5kdc[15669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 09:27:48 dist-test-slave-7vjv krb5kdc[15669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.1@KRBTEST.COM
I20260809 09:27:48.390515 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:48.382018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.9.102.1:33817)
0809 09:27:48.382382 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:48.382391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:48.382411 (+ 20us) server_negotiation.cc:407] Connection header received
0809 09:27:48.382452 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:48.382456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:48.382598 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:48.382735 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:48.383344 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:48.384056 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:48.384937 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:48.385287 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:48.385379 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:48.387496 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:48.387503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:48.387513 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:48.387535 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:48.389003 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:48.389396 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:48.389400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:48.389401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:48.389488 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:48.389719 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:48.389721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:48.389722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:48.389915 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:48.390063 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:48.390190 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 09:27:48.390330 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":150,"threads_started":1}
Aug 09 09:27:48 dist-test-slave-7vjv krb5kdc[15669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:48.399865 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:48.393132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.9.102.62:36343)
0809 09:27:48.393287 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:48.393290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:48.393322 (+ 32us) server_negotiation.cc:407] Connection header received
0809 09:27:48.393404 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:48.393405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:48.393438 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:48.393495 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:48.394014 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:48.394441 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:48.395078 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:48.395245 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:48.395327 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:48.397514 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:48.397517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:48.397518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:48.397536 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:48.398569 (+ 1033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:48.398883 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:48.398885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:48.398886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:48.398946 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:48.399146 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:48.399148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:48.399149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:48.399309 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:48.399397 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:48.399553 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 09:27:48.399663 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260809 09:27:48.400254 15746 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48704
I20260809 09:27:48.400777 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 15838
I20260809 09:27:48.415609 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 16072
I20260809 09:27:48.427356 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 16300
I20260809 09:27:48.432369 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 15697
2026-08-09T09:27:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2148 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:27:48 dist-test-slave-7vjv krb5kdc[16543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:27:48 dist-test-slave-7vjv krb5kdc[16543](info): set up 2 sockets
Aug 09 09:27:48 dist-test-slave-7vjv krb5kdc[16543](info): commencing operation
krb5kdc: starting...
W20260809 09:27:49.585266 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:27:49 dist-test-slave-7vjv krb5kdc[16543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:27:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:27:49.690548 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40607
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:40607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:49.790855 16572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:49.791183 16572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:49.791266 16572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:49.794762 16572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:49.794847 16572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:49.794884 16572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:49.794919 16572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:49.794952 16572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:49.799688 16572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:40607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:49.800747 16572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:49.801649 16572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:49.809418 16582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:49.809496 16572 server_base.cc:1034] running on GCE node
W20260809 09:27:49.809420 16581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:49.809418 16584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:49.810163 16572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:49.811087 16572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:49.812263 16572 hybrid_clock.cc:648] HybridClock initialized: now 1786267669812251 us; error 34 us; skew 500 ppm
Aug 09 09:27:49 dist-test-slave-7vjv krb5kdc[16543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267669, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:49.814637 16572 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:27:49.815287 16572 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:49.815346 16572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:49.815490 16572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:49.817333 16572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "bd0e44cba25042479bf667d592c58f88"
format_stamp: "Formatted at 2026-08-09 09:27:49 on dist-test-slave-7vjv"
server_key: "72572e390dc943d21838037b92be286f"
server_key_iv: "a128823e2d5225bb8d8b5dd38fc9aebd"
server_key_version: "encryptionkey@0"
I20260809 09:27:49.817852 16572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "bd0e44cba25042479bf667d592c58f88"
format_stamp: "Formatted at 2026-08-09 09:27:49 on dist-test-slave-7vjv"
server_key: "72572e390dc943d21838037b92be286f"
server_key_iv: "a128823e2d5225bb8d8b5dd38fc9aebd"
server_key_version: "encryptionkey@0"
I20260809 09:27:49.821029 16572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260809 09:27:49.823055 16592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:49.824185 16572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:27:49.824291 16572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "bd0e44cba25042479bf667d592c58f88"
format_stamp: "Formatted at 2026-08-09 09:27:49 on dist-test-slave-7vjv"
server_key: "72572e390dc943d21838037b92be286f"
server_key_iv: "a128823e2d5225bb8d8b5dd38fc9aebd"
server_key_version: "encryptionkey@0"
I20260809 09:27:49.824368 16572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:49.840360 16572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:49.842777 16572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:49.842900 16572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:49.849303 16572 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:40607
I20260809 09:27:49.849551 16682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:40607 every 8 connection(s)
I20260809 09:27:49.849895 16572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:49.852164 16684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:49.855716 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 16572
I20260809 09:27:49.855856 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:49.856150 9624 external_mini_cluster.cc:1428] Setting key 587d041327e369f832122951b8940245
I20260809 09:27:49.857034 16684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88: Bootstrap starting.
I20260809 09:27:49.859359 16684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:49.860163 16684 log.cc:824] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:49.861892 16684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88: No bootstrap required, opened a new log
Aug 09 09:27:49 dist-test-slave-7vjv krb5kdc[16543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:49.865193 16684 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0e44cba25042479bf667d592c58f88" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40607 } }
I20260809 09:27:49.865451 16684 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:49.865527 16684 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0e44cba25042479bf667d592c58f88, State: Initialized, Role: FOLLOWER
I20260809 09:27:49.866029 16684 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0e44cba25042479bf667d592c58f88" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40607 } }
I20260809 09:27:49.866222 16684 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:49.866285 16684 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:49.866375 16684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:49.867321 16684 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0e44cba25042479bf667d592c58f88" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40607 } }
I20260809 09:27:49.867637 16684 leader_election.cc:302] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd0e44cba25042479bf667d592c58f88; no voters:
I20260809 09:27:49.867970 16684 leader_election.cc:288] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:49.868073 16694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:49.868300 16694 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [term 1 LEADER]: Becoming Leader. State: Replica: bd0e44cba25042479bf667d592c58f88, State: Running, Role: LEADER
I20260809 09:27:49.868575 16694 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0e44cba25042479bf667d592c58f88" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40607 } }
I20260809 09:27:49.869203 16684 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:49.869964 16695 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd0e44cba25042479bf667d592c58f88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0e44cba25042479bf667d592c58f88" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40607 } } }
I20260809 09:27:49.870092 16695 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:49.870401 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:49.857360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.9.102.62:40607)
0809 09:27:49.857707 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:49.857722 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:49.857757 (+ 35us) server_negotiation.cc:407] Connection header received
0809 09:27:49.858498 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:49.858518 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:49.858858 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:49.859165 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:49.859848 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:49.861041 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:49.861833 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:49.862462 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:49.862607 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:49.864806 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:49.864823 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:49.864835 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:49.864861 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:49.866775 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:49.867190 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:49.867197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:49.867205 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:49.867336 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:49.867613 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:49.867616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:49.867618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:49.868097 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:49.869046 (+ 949us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:49.869197 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 09:27:49.869461 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
I20260809 09:27:49.870450 16712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:49.870515 16696 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd0e44cba25042479bf667d592c58f88. Latest consensus state: current_term: 1 leader_uuid: "bd0e44cba25042479bf667d592c58f88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0e44cba25042479bf667d592c58f88" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40607 } } }
I20260809 09:27:49.870784 16696 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:49.873612 16712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:49.878014 16712 catalog_manager.cc:1347] Generated new cluster ID: f33c6515f57a487f99847214a7e5a65a
I20260809 09:27:49.878088 16712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:49.891897 16712 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:49.892711 16712 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:49.898439 16712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88: Generated new TSK 0
I20260809 09:27:49.899056 16712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:27:49.931262 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40607
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:50.033655 16729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:50.033861 16729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:50.033900 16729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:50.037737 16729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:50.037788 16729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:50.037855 16729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:27:50.042503 16729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:50.043280 16729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:50.044111 16729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:50.053508 16739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:50.053529 16738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:50.053509 16742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:50.053861 16729 server_base.cc:1034] running on GCE node
I20260809 09:27:50.054270 16729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:50.054811 16729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:50.055959 16729 hybrid_clock.cc:648] HybridClock initialized: now 1786267670055941 us; error 28 us; skew 500 ppm
Aug 09 09:27:50 dist-test-slave-7vjv krb5kdc[16543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:50.058727 16729 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:27:50.059379 16729 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:50.059437 16729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:50.059706 16729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:50.061599 16729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "083ea7c5c3074b8b84d153381683ac38"
format_stamp: "Formatted at 2026-08-09 09:27:50 on dist-test-slave-7vjv"
server_key: "df3147ff72ac1946f53fad7e0c1087b3"
server_key_iv: "c16a1011864bedb86b77a91a2252d7f8"
server_key_version: "encryptionkey@0"
I20260809 09:27:50.062136 16729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "083ea7c5c3074b8b84d153381683ac38"
format_stamp: "Formatted at 2026-08-09 09:27:50 on dist-test-slave-7vjv"
server_key: "df3147ff72ac1946f53fad7e0c1087b3"
server_key_iv: "c16a1011864bedb86b77a91a2252d7f8"
server_key_version: "encryptionkey@0"
I20260809 09:27:50.065618 16729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260809 09:27:50.067807 16749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:50.069114 16729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:50.069253 16729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "083ea7c5c3074b8b84d153381683ac38"
format_stamp: "Formatted at 2026-08-09 09:27:50 on dist-test-slave-7vjv"
server_key: "df3147ff72ac1946f53fad7e0c1087b3"
server_key_iv: "c16a1011864bedb86b77a91a2252d7f8"
server_key_version: "encryptionkey@0"
I20260809 09:27:50.069371 16729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:50.083601 16729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:50.086004 16729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:50.086191 16729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:50.086742 16729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:50.087460 16729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:50.087510 16729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:50.087574 16729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:50.087603 16729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:50.095877 16729 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:39877
I20260809 09:27:50.095896 16951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:39877 every 8 connection(s)
I20260809 09:27:50.096503 16729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
Aug 09 09:27:50 dist-test-slave-7vjv krb5kdc[16543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:50.106698 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 16729
I20260809 09:27:50.106845 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:27:50.107184 9624 external_mini_cluster.cc:1428] Setting key f51b6dd55886336cdf158754263aad99
I20260809 09:27:50.108433 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.097959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:52665 (local address 127.9.102.62:40607)
0809 09:27:50.098080 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:50.098083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:50.098764 (+ 681us) server_negotiation.cc:407] Connection header received
0809 09:27:50.100042 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:50.100046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:50.100115 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:50.100208 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:50.101184 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.101672 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.102707 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.102935 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.103051 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.105135 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:50.105138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:50.105140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:50.105161 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:50.106407 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:50.107011 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:50.107013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:50.107014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:50.107108 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:50.107465 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:50.107468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:50.107468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:50.107624 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:50.107725 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:50.108171 (+ 446us) server_negotiation.cc:299] Negotiation successful
0809 09:27:50.108284 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:27:50.109097 16961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.098234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40607 (local address 127.9.102.1:52665)
0809 09:27:50.098609 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:50.098642 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:50.099831 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:50.100358 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:50.100364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:50.100664 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:50.101044 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.101056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.101790 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.101793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.102580 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.102589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.103076 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.103080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.103230 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.103648 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:50.103662 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:50.104981 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:50.106518 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:50.106525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:50.106538 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:50.106875 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:50.107199 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:50.107204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:50.107206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:50.107368 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:50.107725 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:50.107732 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:50.107935 (+ 203us) client_negotiation.cc:769] Sending connection context
0809 09:27:50.108097 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 09:27:50.108375 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
I20260809 09:27:50.110332 16952 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40607
I20260809 09:27:50.110569 16952 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:50.110950 16952 heartbeater.cc:511] Master 127.9.102.62:40607 requested a full tablet report, sending...
I20260809 09:27:50.112389 16613 ts_manager.cc:194] Registered new tserver with Master: 083ea7c5c3074b8b84d153381683ac38 (127.9.102.1:39877)
I20260809 09:27:50.113379 16613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:52665
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:27:50.133131 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40607
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:50.248296 16965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:50.248503 16965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:50.248538 16965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:50.252152 16965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:50.252197 16965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:50.252246 16965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:27:50.256582 16965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:50.257370 16965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:50.258190 16965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:50.266564 16975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:50.266580 16973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:50.266573 16971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:50.266907 16965 server_base.cc:1034] running on GCE node
I20260809 09:27:50.267285 16965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:50.267771 16965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:50.268983 16965 hybrid_clock.cc:648] HybridClock initialized: now 1786267670268955 us; error 41 us; skew 500 ppm
Aug 09 09:27:50 dist-test-slave-7vjv krb5kdc[16543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:50.271796 16965 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:27:50.272503 16965 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:50.272603 16965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:50.272792 16965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:50.274622 16965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "1a92b2d308284048a060c1b7620970e2"
format_stamp: "Formatted at 2026-08-09 09:27:50 on dist-test-slave-7vjv"
server_key: "805745562328ea1317f8afe828fa1c51"
server_key_iv: "20f01386fa26fde2d833912bdc3175a7"
server_key_version: "encryptionkey@0"
I20260809 09:27:50.275082 16965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "1a92b2d308284048a060c1b7620970e2"
format_stamp: "Formatted at 2026-08-09 09:27:50 on dist-test-slave-7vjv"
server_key: "805745562328ea1317f8afe828fa1c51"
server_key_iv: "20f01386fa26fde2d833912bdc3175a7"
server_key_version: "encryptionkey@0"
I20260809 09:27:50.278137 16965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:50.280148 16985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:50.281318 16965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:50.281458 16965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "1a92b2d308284048a060c1b7620970e2"
format_stamp: "Formatted at 2026-08-09 09:27:50 on dist-test-slave-7vjv"
server_key: "805745562328ea1317f8afe828fa1c51"
server_key_iv: "20f01386fa26fde2d833912bdc3175a7"
server_key_version: "encryptionkey@0"
I20260809 09:27:50.281560 16965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:50.303794 16965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:50.306227 16965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:50.306439 16965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:50.306937 16965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:50.307727 16965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:50.307817 16965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:50.307880 16965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:50.307919 16965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:50.315984 16965 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:41269
I20260809 09:27:50.316001 17178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:41269 every 8 connection(s)
I20260809 09:27:50.316510 16965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:27:50.318670 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 16965
I20260809 09:27:50.318826 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:27:50.319175 9624 external_mini_cluster.cc:1428] Setting key aa7d6f7c0902c0393dd285c202d0367b
Aug 09 09:27:50 dist-test-slave-7vjv krb5kdc[16543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:50.330245 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.317995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:33931 (local address 127.9.102.62:40607)
0809 09:27:50.318113 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:50.318117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:50.319349 (+ 1232us) server_negotiation.cc:407] Connection header received
0809 09:27:50.320241 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:50.320244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:50.320300 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:50.320382 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:50.321778 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.322248 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.323471 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.323846 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.323971 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.326568 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:50.326575 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:50.326582 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:50.326613 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:50.328347 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:50.328815 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:50.328818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:50.328821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:50.328909 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:50.329218 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:50.329224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:50.329226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:50.329433 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:50.329534 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:50.329950 (+ 416us) server_negotiation.cc:299] Negotiation successful
0809 09:27:50.330089 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
I20260809 09:27:50.330796 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.318298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40607 (local address 127.9.102.2:33931)
0809 09:27:50.319190 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:50.319245 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:50.319983 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:50.320551 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:50.320557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:50.321019 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:50.321627 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.321648 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.322369 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.322375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.323332 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.323342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.324018 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.324025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.324278 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.324824 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:50.324848 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:50.326391 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:50.328474 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:50.328479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:50.328488 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:50.328712 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:50.329005 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:50.329006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:50.329008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:50.329126 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:50.329538 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:50.329542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:50.329728 (+ 186us) client_negotiation.cc:769] Sending connection context
0809 09:27:50.329859 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 09:27:50.330075 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
I20260809 09:27:50.331895 17180 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40607
I20260809 09:27:50.332126 17180 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:50.332574 17180 heartbeater.cc:511] Master 127.9.102.62:40607 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
I20260809 09:27:50.333614 16613 ts_manager.cc:194] Registered new tserver with Master: 1a92b2d308284048a060c1b7620970e2 (127.9.102.2:41269)
I20260809 09:27:50.334122 16613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:33931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:27:50.350700 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40607
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:50.452536 17187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:50.452759 17187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:50.452795 17187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:50.456436 17187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:50.456478 17187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:50.456528 17187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:27:50.461117 17187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:50.461982 17187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:50.462874 17187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:50.471269 17199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:50.471336 17198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:50.471279 17203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:50.471505 17187 server_base.cc:1034] running on GCE node
I20260809 09:27:50.471879 17187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:50.472558 17187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:50.473734 17187 hybrid_clock.cc:648] HybridClock initialized: now 1786267670473727 us; error 38 us; skew 500 ppm
Aug 09 09:27:50 dist-test-slave-7vjv krb5kdc[16543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:50.476298 17187 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:27:50.476897 17187 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:50.476953 17187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:50.477101 17187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:50.478739 17187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "d8a8cf6ecb0347d691946cc1069874a6"
format_stamp: "Formatted at 2026-08-09 09:27:50 on dist-test-slave-7vjv"
server_key: "f3b3eacdb38e014ee5f76a825b79a492"
server_key_iv: "5bef99e2c2850b67d8371fa75bde5942"
server_key_version: "encryptionkey@0"
I20260809 09:27:50.479178 17187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "d8a8cf6ecb0347d691946cc1069874a6"
format_stamp: "Formatted at 2026-08-09 09:27:50 on dist-test-slave-7vjv"
server_key: "f3b3eacdb38e014ee5f76a825b79a492"
server_key_iv: "5bef99e2c2850b67d8371fa75bde5942"
server_key_version: "encryptionkey@0"
I20260809 09:27:50.482467 17187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260809 09:27:50.484560 17216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:50.485736 17187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:50.485854 17187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "d8a8cf6ecb0347d691946cc1069874a6"
format_stamp: "Formatted at 2026-08-09 09:27:50 on dist-test-slave-7vjv"
server_key: "f3b3eacdb38e014ee5f76a825b79a492"
server_key_iv: "5bef99e2c2850b67d8371fa75bde5942"
server_key_version: "encryptionkey@0"
I20260809 09:27:50.485950 17187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:50.503268 17187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:50.505808 17187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:50.505990 17187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:50.506510 17187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:50.507241 17187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:50.507292 17187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:50.507359 17187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:50.507397 17187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:50.515352 17187 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:37331
I20260809 09:27:50.515370 17422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:37331 every 8 connection(s)
I20260809 09:27:50.515847 17187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:27:50.516093 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 17187
I20260809 09:27:50.516225 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:27:50.516511 9624 external_mini_cluster.cc:1428] Setting key d999c0e799a42b64cfdd40a871538eb8
Aug 09 09:27:50 dist-test-slave-7vjv krb5kdc[16543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267670, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:50.527350 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.517469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:47511 (local address 127.9.102.62:40607)
0809 09:27:50.517596 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:50.517599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:50.518438 (+ 839us) server_negotiation.cc:407] Connection header received
0809 09:27:50.519666 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:50.519668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:50.519710 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:50.519771 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:50.520758 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.521259 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.522240 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.522437 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.522501 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.524626 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:50.524629 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:50.524631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:50.524653 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:50.525772 (+ 1119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:50.526210 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:50.526212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:50.526213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:50.526281 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:50.526537 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:50.526539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:50.526540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:50.526700 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:50.526811 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:50.527166 (+ 355us) server_negotiation.cc:299] Negotiation successful
0809 09:27:50.527248 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 09:27:50.528044 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.517859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40607 (local address 127.9.102.3:47511)
0809 09:27:50.518307 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:50.518357 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:50.519484 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:50.519883 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:50.519888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:50.520230 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:50.520638 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.520650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.521362 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.521365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.522145 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.522154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.522544 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.522546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.522662 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.523045 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:50.523058 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:50.524481 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:50.525866 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:50.525870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:50.525879 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:50.526105 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:50.526352 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:50.526354 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:50.526355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:50.526468 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:50.526801 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:50.526805 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:50.526983 (+ 178us) client_negotiation.cc:769] Sending connection context
0809 09:27:50.527122 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 09:27:50.527317 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260809 09:27:50.529294 17423 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40607
I20260809 09:27:50.529568 17423 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:50.530036 17423 heartbeater.cc:511] Master 127.9.102.62:40607 requested a full tablet report, sending...
I20260809 09:27:50.530876 16613 ts_manager.cc:194] Registered new tserver with Master: d8a8cf6ecb0347d691946cc1069874a6 (127.9.102.3:37331)
I20260809 09:27:50.531374 16613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:47511
I20260809 09:27:50.541541 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:27:50.549665 16688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.543289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44132 (local address 127.9.102.62:40607)
0809 09:27:50.543411 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:50.543415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:50.543613 (+ 198us) server_negotiation.cc:407] Connection header received
0809 09:27:50.543707 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:50.543709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:50.543741 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:50.543805 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:50.544468 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.544890 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.545595 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.545769 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.545835 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.546946 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:50.546948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:50.546950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:50.546969 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:50.548334 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:50.548707 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:50.548709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:50.548710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:50.548777 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:50.548997 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:50.548998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:50.548999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:50.549191 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:50.549288 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:50.549478 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 09:27:50.549568 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 09:27:50.551959 16613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:27:50.554548 16613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:27:50.567361 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.562195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.9.102.3:37331)
0809 09:27:50.562700 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:50.562718 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:50.562743 (+ 25us) server_negotiation.cc:407] Connection header received
0809 09:27:50.562820 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:50.562826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:50.563035 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:50.563175 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:50.563609 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.564604 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.566122 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.566709 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.566811 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.566933 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:50.567140 (+ 207us) server_negotiation.cc:299] Negotiation successful
0809 09:27:50.567224 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":135,"threads_started":1}
I20260809 09:27:50.567660 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.562032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:37331 (local address 127.0.0.1:38540)
0809 09:27:50.562361 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:50.562411 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:50.562528 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:50.563340 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:50.563347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:50.563370 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:50.563476 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.563484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.564764 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.564767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.566006 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.566015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.566816 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.566820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.566936 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.566977 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 09:27:50.567100 (+ 123us) client_negotiation.cc:240] Negotiation successful
0809 09:27:50.567493 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":36,"thread_start_us":96,"threads_started":1}
I20260809 09:27:50.568114 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.562518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:39877 (local address 127.0.0.1:51748)
0809 09:27:50.563450 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:50.563461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:50.563532 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:50.564000 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:50.564003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:50.564022 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:50.564143 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.564150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.565434 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.565437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.566593 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.566612 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.567670 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.567673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.567776 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.567808 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 09:27:50.567875 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 09:27:50.567943 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260809 09:27:50.568186 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.562992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.9.102.1:39877)
0809 09:27:50.563668 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:50.563675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:50.563690 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:27:50.563744 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:50.563748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:50.563868 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:50.563966 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:50.564275 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.565296 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.566725 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.567550 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.567684 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.567832 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:50.567899 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 09:27:50.568000 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":145,"threads_started":1}
I20260809 09:27:50.569581 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.562698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:41269 (local address 127.0.0.1:55064)
0809 09:27:50.564199 (+ 1501us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:50.564225 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:50.564335 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:50.564967 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:50.564969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:50.564985 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:50.565061 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.565065 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.566579 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.566583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.568084 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.568094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.569221 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.569224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.569316 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.569336 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 09:27:50.569409 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 09:27:50.569468 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1442,"mutex_wait_us":441,"thread_start_us":96,"threads_started":1}
I20260809 09:27:50.569721 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.563118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.9.102.2:41269)
0809 09:27:50.563477 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:50.563488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:50.564351 (+ 863us) server_negotiation.cc:407] Connection header received
0809 09:27:50.564447 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:50.564452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:50.564614 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:50.564776 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:50.565196 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.566415 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.568255 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.569084 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.569191 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.569330 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:50.569448 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 09:27:50.569536 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":89,"threads_started":1}
I20260809 09:27:50.570753 16829 tablet_service.cc:1467] Processing CreateTablet for tablet fe26afbb31ac406cb96ebdbb1faa218a (DEFAULT_TABLE table=test-table [id=ed15b12ced744f01ab03f4485f513c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:50.570658 17302 tablet_service.cc:1467] Processing CreateTablet for tablet fe26afbb31ac406cb96ebdbb1faa218a (DEFAULT_TABLE table=test-table [id=ed15b12ced744f01ab03f4485f513c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:50.571663 16829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe26afbb31ac406cb96ebdbb1faa218a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:50.571696 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe26afbb31ac406cb96ebdbb1faa218a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:50.572034 17074 tablet_service.cc:1467] Processing CreateTablet for tablet fe26afbb31ac406cb96ebdbb1faa218a (DEFAULT_TABLE table=test-table [id=ed15b12ced744f01ab03f4485f513c23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:50.572908 17074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe26afbb31ac406cb96ebdbb1faa218a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:50.577984 17451 tablet_bootstrap.cc:492] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38: Bootstrap starting.
I20260809 09:27:50.579253 17452 tablet_bootstrap.cc:492] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6: Bootstrap starting.
I20260809 09:27:50.580228 17451 tablet_bootstrap.cc:654] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:50.580411 17453 tablet_bootstrap.cc:492] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2: Bootstrap starting.
I20260809 09:27:50.581094 17452 tablet_bootstrap.cc:654] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:50.581559 17451 log.cc:824] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:50.582295 17452 log.cc:824] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:50.582508 17453 tablet_bootstrap.cc:654] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:50.583333 17453 log.cc:824] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:50.583356 17451 tablet_bootstrap.cc:492] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38: No bootstrap required, opened a new log
I20260809 09:27:50.583556 17451 ts_tablet_manager.cc:1397] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260809 09:27:50.585148 17452 tablet_bootstrap.cc:492] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6: No bootstrap required, opened a new log
I20260809 09:27:50.585331 17452 ts_tablet_manager.cc:1397] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 09:27:50.585353 17453 tablet_bootstrap.cc:492] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2: No bootstrap required, opened a new log
I20260809 09:27:50.585532 17453 ts_tablet_manager.cc:1397] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 09:27:50.587581 17451 raft_consensus.cc:348] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a8cf6ecb0347d691946cc1069874a6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 37331 } } peers { permanent_uuid: "1a92b2d308284048a060c1b7620970e2" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41269 } } peers { permanent_uuid: "083ea7c5c3074b8b84d153381683ac38" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39877 } }
I20260809 09:27:50.587855 17451 raft_consensus.cc:374] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:50.587918 17451 raft_consensus.cc:729] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083ea7c5c3074b8b84d153381683ac38, State: Initialized, Role: FOLLOWER
I20260809 09:27:50.588389 17451 consensus_queue.cc:260] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a8cf6ecb0347d691946cc1069874a6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 37331 } } peers { permanent_uuid: "1a92b2d308284048a060c1b7620970e2" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41269 } } peers { permanent_uuid: "083ea7c5c3074b8b84d153381683ac38" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39877 } }
I20260809 09:27:50.588400 17452 raft_consensus.cc:348] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a8cf6ecb0347d691946cc1069874a6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 37331 } } peers { permanent_uuid: "1a92b2d308284048a060c1b7620970e2" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41269 } } peers { permanent_uuid: "083ea7c5c3074b8b84d153381683ac38" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39877 } }
I20260809 09:27:50.588668 17452 raft_consensus.cc:374] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:50.588801 17452 raft_consensus.cc:729] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a8cf6ecb0347d691946cc1069874a6, State: Initialized, Role: FOLLOWER
I20260809 09:27:50.589087 16952 heartbeater.cc:503] Master 127.9.102.62:40607 was elected leader, sending a full tablet report...
I20260809 09:27:50.589303 17452 consensus_queue.cc:260] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a8cf6ecb0347d691946cc1069874a6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 37331 } } peers { permanent_uuid: "1a92b2d308284048a060c1b7620970e2" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41269 } } peers { permanent_uuid: "083ea7c5c3074b8b84d153381683ac38" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39877 } }
I20260809 09:27:50.589457 17453 raft_consensus.cc:348] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a8cf6ecb0347d691946cc1069874a6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 37331 } } peers { permanent_uuid: "1a92b2d308284048a060c1b7620970e2" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41269 } } peers { permanent_uuid: "083ea7c5c3074b8b84d153381683ac38" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39877 } }
I20260809 09:27:50.589720 17453 raft_consensus.cc:374] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:50.589797 17453 raft_consensus.cc:729] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a92b2d308284048a060c1b7620970e2, State: Initialized, Role: FOLLOWER
I20260809 09:27:50.590018 17423 heartbeater.cc:503] Master 127.9.102.62:40607 was elected leader, sending a full tablet report...
I20260809 09:27:50.590180 17453 consensus_queue.cc:260] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a8cf6ecb0347d691946cc1069874a6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 37331 } } peers { permanent_uuid: "1a92b2d308284048a060c1b7620970e2" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41269 } } peers { permanent_uuid: "083ea7c5c3074b8b84d153381683ac38" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39877 } }
I20260809 09:27:50.590317 17452 ts_tablet_manager.cc:1428] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:27:50.591029 17180 heartbeater.cc:503] Master 127.9.102.62:40607 was elected leader, sending a full tablet report...
I20260809 09:27:50.591086 17453 ts_tablet_manager.cc:1428] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260809 09:27:50.591784 17451 ts_tablet_manager.cc:1428] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
W20260809 09:27:50.597432 16957 tablet.cc:2369] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:27:50.766932 17424 tablet.cc:2369] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:27:50.778214 17463 raft_consensus.cc:482] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:27:50.778378 17463 raft_consensus.cc:504] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a8cf6ecb0347d691946cc1069874a6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 37331 } } peers { permanent_uuid: "1a92b2d308284048a060c1b7620970e2" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41269 } } peers { permanent_uuid: "083ea7c5c3074b8b84d153381683ac38" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39877 } }
I20260809 09:27:50.779438 17463 leader_election.cc:288] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8a8cf6ecb0347d691946cc1069874a6 (127.9.102.3:37331), 083ea7c5c3074b8b84d153381683ac38 (127.9.102.1:39877)
I20260809 09:27:50.783205 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.779687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:37331 (local address 127.9.102.2:48157)
0809 09:27:50.779870 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:50.779885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:50.779982 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:50.780307 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:50.780312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:50.780332 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:50.780458 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.780468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.781268 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.781271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.782108 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.782118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.782777 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.782783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.782883 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.782911 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 09:27:50.782980 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 09:27:50.783072 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260809 09:27:50.783215 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.779762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:48157 (local address 127.9.102.3:37331)
0809 09:27:50.779885 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:50.779889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:50.779968 (+ 79us) server_negotiation.cc:407] Connection header received
0809 09:27:50.780123 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:50.780126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:50.780181 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:50.780276 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:50.780594 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.781162 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.782243 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.782660 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.782740 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.782823 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:50.783020 (+ 197us) server_negotiation.cc:299] Negotiation successful
0809 09:27:50.783100 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 09:27:50.783254 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.779707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:39877 (local address 127.9.102.2:60947)
0809 09:27:50.780186 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:50.780200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:50.780296 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:50.780525 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:50.780527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:50.780570 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:50.780682 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.780688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.781413 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.781416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.782162 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:50.782201 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.782841 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.782847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:50.782936 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.782954 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 09:27:50.783059 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 09:27:50.783123 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":109,"threads_started":1}
I20260809 09:27:50.783264 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.779768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:60947 (local address 127.9.102.1:39877)
0809 09:27:50.779893 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:50.779900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:50.780286 (+ 386us) server_negotiation.cc:407] Connection header received
0809 09:27:50.780376 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:50.780379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:50.780429 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:50.780510 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:50.780790 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.781306 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.782312 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.782694 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.782822 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.782884 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:50.783079 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 09:27:50.783153 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 09:27:50.783829 17335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe26afbb31ac406cb96ebdbb1faa218a" candidate_uuid: "1a92b2d308284048a060c1b7620970e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8a8cf6ecb0347d691946cc1069874a6" is_pre_election: true
I20260809 09:27:50.783835 16871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe26afbb31ac406cb96ebdbb1faa218a" candidate_uuid: "1a92b2d308284048a060c1b7620970e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083ea7c5c3074b8b84d153381683ac38" is_pre_election: true
I20260809 09:27:50.784099 17335 raft_consensus.cc:2454] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a92b2d308284048a060c1b7620970e2 in term 0.
I20260809 09:27:50.784152 16871 raft_consensus.cc:2454] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a92b2d308284048a060c1b7620970e2 in term 0.
I20260809 09:27:50.784655 16987 leader_election.cc:302] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083ea7c5c3074b8b84d153381683ac38, 1a92b2d308284048a060c1b7620970e2; no voters:
I20260809 09:27:50.784930 17463 raft_consensus.cc:2781] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:27:50.785017 17463 raft_consensus.cc:482] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:27:50.785069 17463 raft_consensus.cc:3037] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:50.786104 17463 raft_consensus.cc:504] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a8cf6ecb0347d691946cc1069874a6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 37331 } } peers { permanent_uuid: "1a92b2d308284048a060c1b7620970e2" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41269 } } peers { permanent_uuid: "083ea7c5c3074b8b84d153381683ac38" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39877 } }
I20260809 09:27:50.786444 17463 leader_election.cc:288] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [CANDIDATE]: Term 1 election: Requested vote from peers d8a8cf6ecb0347d691946cc1069874a6 (127.9.102.3:37331), 083ea7c5c3074b8b84d153381683ac38 (127.9.102.1:39877)
I20260809 09:27:50.786746 17335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe26afbb31ac406cb96ebdbb1faa218a" candidate_uuid: "1a92b2d308284048a060c1b7620970e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8a8cf6ecb0347d691946cc1069874a6"
I20260809 09:27:50.786803 16871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe26afbb31ac406cb96ebdbb1faa218a" candidate_uuid: "1a92b2d308284048a060c1b7620970e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083ea7c5c3074b8b84d153381683ac38"
I20260809 09:27:50.786883 17335 raft_consensus.cc:3037] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:50.786918 16871 raft_consensus.cc:3037] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:50.787905 17335 raft_consensus.cc:2454] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a92b2d308284048a060c1b7620970e2 in term 1.
I20260809 09:27:50.787930 16871 raft_consensus.cc:2454] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a92b2d308284048a060c1b7620970e2 in term 1.
I20260809 09:27:50.788203 16987 leader_election.cc:302] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083ea7c5c3074b8b84d153381683ac38, 1a92b2d308284048a060c1b7620970e2; no voters:
I20260809 09:27:50.788391 17463 raft_consensus.cc:2781] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:50.788553 17463 raft_consensus.cc:686] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 1 LEADER]: Becoming Leader. State: Replica: 1a92b2d308284048a060c1b7620970e2, State: Running, Role: LEADER
I20260809 09:27:50.788829 17463 consensus_queue.cc:237] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a8cf6ecb0347d691946cc1069874a6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 37331 } } peers { permanent_uuid: "1a92b2d308284048a060c1b7620970e2" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41269 } } peers { permanent_uuid: "083ea7c5c3074b8b84d153381683ac38" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39877 } }
I20260809 09:27:50.791822 16612 catalog_manager.cc:5495] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a92b2d308284048a060c1b7620970e2 (127.9.102.2). New cstate: current_term: 1 leader_uuid: "1a92b2d308284048a060c1b7620970e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8a8cf6ecb0347d691946cc1069874a6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 37331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a92b2d308284048a060c1b7620970e2" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "083ea7c5c3074b8b84d153381683ac38" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39877 } health_report { overall_health: UNKNOWN } } }
W20260809 09:27:50.817430 17181 tablet.cc:2369] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:27:50.858424 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.854828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.9.102.2:41269)
0809 09:27:50.854950 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:50.854954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:50.854968 (+ 14us) server_negotiation.cc:407] Connection header received
0809 09:27:50.855094 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:50.855097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:50.855152 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:50.855256 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:50.855458 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.856194 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.856992 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:50.857297 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:50.857392 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:50.857646 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:50.858078 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:50.858184 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:50.858238 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 09:27:50.858288 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:27:50.863737 16871 raft_consensus.cc:1261] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38 [term 1 FOLLOWER]: Refusing update from remote peer 1a92b2d308284048a060c1b7620970e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:50.863761 17335 raft_consensus.cc:1261] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6 [term 1 FOLLOWER]: Refusing update from remote peer 1a92b2d308284048a060c1b7620970e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:50.864388 17468 consensus_queue.cc:1035] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8a8cf6ecb0347d691946cc1069874a6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 37331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:50.864524 17463 consensus_queue.cc:1035] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083ea7c5c3074b8b84d153381683ac38" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:50.871687 17472 mvcc.cc:205] Tried to move back new op lower bound from 7316552379851911168 to 7316552379553263616. Current Snapshot: MvccSnapshot[applied={T|T < 7316552379851911168}]
I20260809 09:27:50.875407 17475 mvcc.cc:205] Tried to move back new op lower bound from 7316552379851911168 to 7316552379553263616. Current Snapshot: MvccSnapshot[applied={T|T < 7316552379851911168 or (T in {7316552379851911168})}]
I20260809 09:27:50.880529 16613 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:50.880693 16613 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:50.887516 16613 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd0e44cba25042479bf667d592c58f88: Sending DeleteTablet for 3 replicas of tablet fe26afbb31ac406cb96ebdbb1faa218a
I20260809 09:27:50.888341 17074 tablet_service.cc:1514] Processing DeleteTablet for tablet fe26afbb31ac406cb96ebdbb1faa218a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:50 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:55064
I20260809 09:27:50.889632 16829 tablet_service.cc:1514] Processing DeleteTablet for tablet fe26afbb31ac406cb96ebdbb1faa218a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:50 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:51748
I20260809 09:27:50.889694 17489 tablet_replica.cc:330] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2: stopping tablet replica
I20260809 09:27:50.890044 17490 tablet_replica.cc:330] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38: stopping tablet replica
I20260809 09:27:50.890007 17302 tablet_service.cc:1514] Processing DeleteTablet for tablet fe26afbb31ac406cb96ebdbb1faa218a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:50 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:38540
I20260809 09:27:50.890053 17489 raft_consensus.cc:2229] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:27:50.890313 17490 raft_consensus.cc:2229] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:50.890544 17489 raft_consensus.cc:2258] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:50.890655 17490 raft_consensus.cc:2258] T fe26afbb31ac406cb96ebdbb1faa218a P 083ea7c5c3074b8b84d153381683ac38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:50.891170 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 16729
I20260809 09:27:50.896649 17489 ts_tablet_manager.cc:1905] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:50.899170 17489 ts_tablet_manager.cc:1918] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:27:50.899295 17489 log.cc:1196] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/fe26afbb31ac406cb96ebdbb1faa218a
I20260809 09:27:50.899904 17489 ts_tablet_manager.cc:1939] T fe26afbb31ac406cb96ebdbb1faa218a P 1a92b2d308284048a060c1b7620970e2: Deleting consensus metadata
I20260809 09:27:50.901125 16593 catalog_manager.cc:4841] TS 1a92b2d308284048a060c1b7620970e2 (127.9.102.2:41269): tablet fe26afbb31ac406cb96ebdbb1faa218a (table test-table [id=ed15b12ced744f01ab03f4485f513c23]) successfully deleted
I20260809 09:27:50.904273 17491 tablet_replica.cc:330] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6: stopping tablet replica
I20260809 09:27:50.904524 17491 raft_consensus.cc:2229] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:50.904805 17491 raft_consensus.cc:2258] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:50.905953 17491 ts_tablet_manager.cc:1905] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 09:27:50.906793 16594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:39877 (error 108)
I20260809 09:27:50.907619 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 16965
I20260809 09:27:50.907831 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:50.907524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:39877 (local address 127.0.0.1:51756)
0809 09:27:50.907654 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:50.907735 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 09:27:50.908607 17491 ts_tablet_manager.cc:1918] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:27:50.908694 17491 log.cc:1196] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/fe26afbb31ac406cb96ebdbb1faa218a
I20260809 09:27:50.908963 17491 ts_tablet_manager.cc:1939] T fe26afbb31ac406cb96ebdbb1faa218a P d8a8cf6ecb0347d691946cc1069874a6: Deleting consensus metadata
W20260809 09:27:50.909876 16594 catalog_manager.cc:4584] TS 083ea7c5c3074b8b84d153381683ac38 (127.9.102.1:39877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe26afbb31ac406cb96ebdbb1faa218a: Network error: Client connection negotiation failed: client connection to 127.9.102.1:39877: connect: Connection refused (error 111)
I20260809 09:27:50.914067 16593 catalog_manager.cc:4841] TS d8a8cf6ecb0347d691946cc1069874a6 (127.9.102.3:37331): tablet fe26afbb31ac406cb96ebdbb1faa218a (table test-table [id=ed15b12ced744f01ab03f4485f513c23]) successfully deleted
I20260809 09:27:50.918499 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 17187
I20260809 09:27:50.935971 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 16572
2026-08-09T09:27:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2494 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T09:27:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:50Z Disabled control of system clock
I20260809 09:27:50.967252 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39521
--webserver_interface=127.9.102.62
--webserver_port=0
--builtin_ntp_servers=127.9.102.20:37619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 09:27:51.068214 17498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:51.068416 17498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:51.068471 17498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:51.071974 17498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:51.072028 17498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:51.072043 17498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:51.072057 17498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:51.072124 17498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:51.076527 17498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:37619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39521
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:51.077401 17498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:51.078619 17498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:51.086112 17506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:51.086126 17513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:51.086112 17509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:51.086236 17498 server_base.cc:1034] running on GCE node
I20260809 09:27:51.086882 17498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:51.087806 17498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:51.088958 17498 hybrid_clock.cc:648] HybridClock initialized: now 1786267671088915 us; error 56 us; skew 500 ppm
I20260809 09:27:51.090580 17498 webserver.cc:460] Webserver started at http://127.9.102.62:43749/ using document root <none> and password file <none>
I20260809 09:27:51.091081 17498 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:51.091157 17498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:51.091383 17498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:51.093235 17498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "968e31debd2243df8c8d3ac3f8ef9399"
format_stamp: "Formatted at 2026-08-09 09:27:51 on dist-test-slave-7vjv"
server_key: "6cff7d2620ca53a001a7c25d2380b417"
server_key_iv: "4e3117f4a980ef66b19d9e0b2c9b8fde"
server_key_version: "encryptionkey@0"
I20260809 09:27:51.093751 17498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "968e31debd2243df8c8d3ac3f8ef9399"
format_stamp: "Formatted at 2026-08-09 09:27:51 on dist-test-slave-7vjv"
server_key: "6cff7d2620ca53a001a7c25d2380b417"
server_key_iv: "4e3117f4a980ef66b19d9e0b2c9b8fde"
server_key_version: "encryptionkey@0"
I20260809 09:27:51.097083 17498 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:27:51.099299 17525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:51.100521 17498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:27:51.100651 17498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "968e31debd2243df8c8d3ac3f8ef9399"
format_stamp: "Formatted at 2026-08-09 09:27:51 on dist-test-slave-7vjv"
server_key: "6cff7d2620ca53a001a7c25d2380b417"
server_key_iv: "4e3117f4a980ef66b19d9e0b2c9b8fde"
server_key_version: "encryptionkey@0"
I20260809 09:27:51.100754 17498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:51.104787 17498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:51.105345 17498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:51.105525 17498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:51.111490 17498 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:39521
I20260809 09:27:51.111493 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:39521 every 8 connection(s)
I20260809 09:27:51.112365 17498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:51.115020 17613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:51.119608 17613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399: Bootstrap starting.
I20260809 09:27:51.121770 17613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:51.122102 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 17498
I20260809 09:27:51.122220 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:51.122448 9624 external_mini_cluster.cc:1428] Setting key 46d5570c0ae0798a2b8de87709aa9e3d
I20260809 09:27:51.122493 17613 log.cc:824] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:51.124740 17613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399: No bootstrap required, opened a new log
I20260809 09:27:51.127108 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.123398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.9.102.62:39521)
0809 09:27:51.123926 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:51.123945 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:51.123977 (+ 32us) server_negotiation.cc:407] Connection header received
0809 09:27:51.124623 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:51.124641 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:51.124951 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:51.125203 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:51.125980 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:51.125984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:51.125994 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:51.126003 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:51.126053 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:51.126068 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:51.126146 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:51.126277 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 09:27:51.126496 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":116,"threads_started":1}
I20260809 09:27:51.128611 17613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968e31debd2243df8c8d3ac3f8ef9399" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39521 } }
I20260809 09:27:51.128849 17613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:51.128909 17613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968e31debd2243df8c8d3ac3f8ef9399, State: Initialized, Role: FOLLOWER
I20260809 09:27:51.129356 17613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968e31debd2243df8c8d3ac3f8ef9399" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39521 } }
I20260809 09:27:51.129498 17613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:51.129563 17613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:51.129684 17613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:51.130450 17613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968e31debd2243df8c8d3ac3f8ef9399" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39521 } }
I20260809 09:27:51.130777 17613 leader_election.cc:302] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 968e31debd2243df8c8d3ac3f8ef9399; no voters:
I20260809 09:27:51.131064 17613 leader_election.cc:288] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:51.131211 17622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:51.131682 17622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [term 1 LEADER]: Becoming Leader. State: Replica: 968e31debd2243df8c8d3ac3f8ef9399, State: Running, Role: LEADER
I20260809 09:27:51.131975 17622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968e31debd2243df8c8d3ac3f8ef9399" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39521 } }
I20260809 09:27:51.132309 17613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:51.133600 17624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 968e31debd2243df8c8d3ac3f8ef9399. Latest consensus state: current_term: 1 leader_uuid: "968e31debd2243df8c8d3ac3f8ef9399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968e31debd2243df8c8d3ac3f8ef9399" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39521 } } }
I20260809 09:27:51.133620 17623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "968e31debd2243df8c8d3ac3f8ef9399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968e31debd2243df8c8d3ac3f8ef9399" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39521 } } }
I20260809 09:27:51.133711 17624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:51.133711 17623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:51.134096 17637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:51.136606 17637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:51.141630 17637 catalog_manager.cc:1347] Generated new cluster ID: 9f7b044459684c99914fb9cfdda6f203
I20260809 09:27:51.141695 17637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:51.163810 17637 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:51.164080 17637 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:51.175064 17637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399: Generated new TSK 0
I20260809 09:27:51.175594 17637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:27:51.183386 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39521
--builtin_ntp_servers=127.9.102.20:37619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 09:27:51.302038 17657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:51.302240 17657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:51.302301 17657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:51.306151 17657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:51.306193 17657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:51.306281 17657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:27:51.310452 17657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:37619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:51.311370 17657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:51.312642 17657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:51.321252 17670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:51.321265 17665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:51.321269 17673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:51.321600 17657 server_base.cc:1034] running on GCE node
I20260809 09:27:51.321971 17657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:51.322528 17657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:51.323701 17657 hybrid_clock.cc:648] HybridClock initialized: now 1786267671323679 us; error 38 us; skew 500 ppm
I20260809 09:27:51.325284 17657 webserver.cc:460] Webserver started at http://127.9.102.1:37163/ using document root <none> and password file <none>
I20260809 09:27:51.325791 17657 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:51.325879 17657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:51.326087 17657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:51.327921 17657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "8f1884cfac75438f9d69555de30eddbc"
format_stamp: "Formatted at 2026-08-09 09:27:51 on dist-test-slave-7vjv"
server_key: "27da7a56567eb9ca2b694ece6f681163"
server_key_iv: "1c4172cb6223d6b046256f5e925ad90b"
server_key_version: "encryptionkey@0"
I20260809 09:27:51.328497 17657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "8f1884cfac75438f9d69555de30eddbc"
format_stamp: "Formatted at 2026-08-09 09:27:51 on dist-test-slave-7vjv"
server_key: "27da7a56567eb9ca2b694ece6f681163"
server_key_iv: "1c4172cb6223d6b046256f5e925ad90b"
server_key_version: "encryptionkey@0"
I20260809 09:27:51.331894 17657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:27:51.334177 17686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:51.335295 17657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:51.335405 17657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "8f1884cfac75438f9d69555de30eddbc"
format_stamp: "Formatted at 2026-08-09 09:27:51 on dist-test-slave-7vjv"
server_key: "27da7a56567eb9ca2b694ece6f681163"
server_key_iv: "1c4172cb6223d6b046256f5e925ad90b"
server_key_version: "encryptionkey@0"
I20260809 09:27:51.335493 17657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:51.339357 17657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:51.339879 17657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:51.340063 17657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:51.340600 17657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:51.341413 17657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:51.341457 17657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:51.341519 17657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:51.341549 17657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:51.349476 17657 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:36599
I20260809 09:27:51.349495 17868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:36599 every 8 connection(s)
I20260809 09:27:51.350179 17657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:27:51.355345 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.351748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:39847 (local address 127.9.102.62:39521)
0809 09:27:51.351872 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:51.351875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:51.352779 (+ 904us) server_negotiation.cc:407] Connection header received
0809 09:27:51.353928 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:51.353931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:51.353973 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:51.354072 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:51.354924 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:51.354927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:51.354929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:51.354940 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:51.354965 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:51.354974 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:51.355065 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:51.355187 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 09:27:51.355230 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 09:27:51.355996 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.352044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39521 (local address 127.9.102.1:39847)
0809 09:27:51.352658 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:51.352688 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:51.353740 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:51.354205 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:51.354211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:51.354282 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:51.354706 (+ 424us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:27:51.354707 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:51.354738 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:27:51.354793 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:51.355045 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:51.355053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:51.355058 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 09:27:51.355212 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 09:27:51.355403 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":111,"threads_started":1}
I20260809 09:27:51.357018 17872 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39521
I20260809 09:27:51.357241 17872 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:51.357568 17872 heartbeater.cc:511] Master 127.9.102.62:39521 requested a full tablet report, sending...
I20260809 09:27:51.358881 17551 ts_manager.cc:194] Registered new tserver with Master: 8f1884cfac75438f9d69555de30eddbc (127.9.102.1:36599)
I20260809 09:27:51.359447 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 17657
I20260809 09:27:51.359570 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:27:51.359824 9624 external_mini_cluster.cc:1428] Setting key 0df0507c7c5493e0014364e445423b49
I20260809 09:27:51.361522 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39521
--builtin_ntp_servers=127.9.102.20:37619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 09:27:51.462045 17876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:51.462252 17876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:51.462286 17876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:51.466558 17876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:51.466606 17876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:51.466686 17876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:27:51.471105 17876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:37619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:51.472059 17876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:51.473328 17876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:51.482398 17890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:51.482421 17886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:51.482398 17884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:51.482676 17876 server_base.cc:1034] running on GCE node
I20260809 09:27:51.483028 17876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:51.483693 17876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:51.484850 17876 hybrid_clock.cc:648] HybridClock initialized: now 1786267671484835 us; error 27 us; skew 500 ppm
I20260809 09:27:51.486541 17876 webserver.cc:460] Webserver started at http://127.9.102.2:38193/ using document root <none> and password file <none>
I20260809 09:27:51.487126 17876 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:51.487207 17876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:51.487452 17876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:51.489660 17876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "8d2b7a619d0d4125ac652d18189f289f"
format_stamp: "Formatted at 2026-08-09 09:27:51 on dist-test-slave-7vjv"
server_key: "34befd79a003c117648cbee86916adca"
server_key_iv: "0ab882e0bc2c582d2181e44955ad3d8d"
server_key_version: "encryptionkey@0"
I20260809 09:27:51.490278 17876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "8d2b7a619d0d4125ac652d18189f289f"
format_stamp: "Formatted at 2026-08-09 09:27:51 on dist-test-slave-7vjv"
server_key: "34befd79a003c117648cbee86916adca"
server_key_iv: "0ab882e0bc2c582d2181e44955ad3d8d"
server_key_version: "encryptionkey@0"
I20260809 09:27:51.493885 17876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:51.496228 17900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:51.497413 17876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:51.497560 17876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "8d2b7a619d0d4125ac652d18189f289f"
format_stamp: "Formatted at 2026-08-09 09:27:51 on dist-test-slave-7vjv"
server_key: "34befd79a003c117648cbee86916adca"
server_key_iv: "0ab882e0bc2c582d2181e44955ad3d8d"
server_key_version: "encryptionkey@0"
I20260809 09:27:51.497674 17876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:51.502266 17876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:51.502854 17876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:51.503067 17876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:51.503645 17876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:51.504652 17876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:51.504725 17876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:51.504791 17876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:51.504849 17876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:51.513777 17876 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:41503
I20260809 09:27:51.513798 18084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:41503 every 8 connection(s)
I20260809 09:27:51.514566 17876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:27:51.517767 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 17876
I20260809 09:27:51.517912 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:27:51.518217 9624 external_mini_cluster.cc:1428] Setting key 1e94d7538a29eb3d4ea694c2433c87e0
I20260809 09:27:51.520154 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39521
--builtin_ntp_servers=127.9.102.20:37619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 09:27:51.520741 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.516355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:52659 (local address 127.9.102.62:39521)
0809 09:27:51.516491 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:51.516495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:51.517441 (+ 946us) server_negotiation.cc:407] Connection header received
0809 09:27:51.518951 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:51.518955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:51.518999 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:51.519126 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:51.520257 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:51.520261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:51.520264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:51.520276 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:51.520299 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:51.520312 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:51.520417 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:51.520561 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 09:27:51.520606 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 09:27:51.521469 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.516717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39521 (local address 127.9.102.2:52659)
0809 09:27:51.517289 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:51.517325 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:51.518753 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:51.519274 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:51.519280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:51.519365 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:51.519971 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:27:51.519972 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:51.520008 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:27:51.520068 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:51.520409 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:51.520418 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:51.520426 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 09:27:51.520600 (+ 174us) client_negotiation.cc:240] Negotiation successful
0809 09:27:51.520804 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260809 09:27:51.522550 18086 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39521
I20260809 09:27:51.522804 18086 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:51.523168 18086 heartbeater.cc:511] Master 127.9.102.62:39521 requested a full tablet report, sending...
I20260809 09:27:51.524232 17551 ts_manager.cc:194] Registered new tserver with Master: 8d2b7a619d0d4125ac652d18189f289f (127.9.102.2:41503)
W20260809 09:27:51.626931 18090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:51.627171 18090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:51.627239 18090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:51.630851 18090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:51.630932 18090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:51.631013 18090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:27:51.635915 18090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:37619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:51.636878 18090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:51.638134 18090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:51.646999 18098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:51.647037 18097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:51.647042 18103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:51.647173 18090 server_base.cc:1034] running on GCE node
I20260809 09:27:51.647675 18090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:51.648277 18090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:51.649466 18090 hybrid_clock.cc:648] HybridClock initialized: now 1786267671649444 us; error 39 us; skew 500 ppm
I20260809 09:27:51.651160 18090 webserver.cc:460] Webserver started at http://127.9.102.3:43253/ using document root <none> and password file <none>
I20260809 09:27:51.651698 18090 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:51.651799 18090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:51.651999 18090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:51.653861 18090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "d91445dbe6ef43a6912df6cd05382268"
format_stamp: "Formatted at 2026-08-09 09:27:51 on dist-test-slave-7vjv"
server_key: "eaab066d4cca7fd05b752af5fbc299c7"
server_key_iv: "be864c35d3e2dc563aa9a6d4c4831ee5"
server_key_version: "encryptionkey@0"
I20260809 09:27:51.654408 18090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "d91445dbe6ef43a6912df6cd05382268"
format_stamp: "Formatted at 2026-08-09 09:27:51 on dist-test-slave-7vjv"
server_key: "eaab066d4cca7fd05b752af5fbc299c7"
server_key_iv: "be864c35d3e2dc563aa9a6d4c4831ee5"
server_key_version: "encryptionkey@0"
I20260809 09:27:51.657842 18090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:51.659886 18109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:51.661005 18090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:51.661141 18090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "d91445dbe6ef43a6912df6cd05382268"
format_stamp: "Formatted at 2026-08-09 09:27:51 on dist-test-slave-7vjv"
server_key: "eaab066d4cca7fd05b752af5fbc299c7"
server_key_iv: "be864c35d3e2dc563aa9a6d4c4831ee5"
server_key_version: "encryptionkey@0"
I20260809 09:27:51.661278 18090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:51.665392 18090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:51.665889 18090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:51.666080 18090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:51.666623 18090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:51.667485 18090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:51.667552 18090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:51.667639 18090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:51.667696 18090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:51.676450 18090 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:46679
I20260809 09:27:51.676473 18297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:46679 every 8 connection(s)
I20260809 09:27:51.677224 18090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:27:51.682580 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.678881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:48153 (local address 127.9.102.62:39521)
0809 09:27:51.679008 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:51.679012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:51.680006 (+ 994us) server_negotiation.cc:407] Connection header received
0809 09:27:51.681184 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:51.681186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:51.681221 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:51.681291 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:51.682278 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:51.682280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:51.682281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:51.682289 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:51.682303 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:51.682309 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:51.682367 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:51.682452 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 09:27:51.682485 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:27:51.683279 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.679258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39521 (local address 127.9.102.3:48153)
0809 09:27:51.679864 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:51.679911 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:51.681008 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:51.681435 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:51.681441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:51.681518 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:51.682070 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:27:51.682071 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:51.682105 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:27:51.682163 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:51.682357 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:51.682361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:51.682364 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 09:27:51.682468 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 09:27:51.682631 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":111,"threads_started":1}
I20260809 09:27:51.684279 18298 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39521
I20260809 09:27:51.684482 18298 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:51.684834 18298 heartbeater.cc:511] Master 127.9.102.62:39521 requested a full tablet report, sending...
I20260809 09:27:51.685734 17551 ts_manager.cc:194] Registered new tserver with Master: d91445dbe6ef43a6912df6cd05382268 (127.9.102.3:46679)
I20260809 09:27:51.686918 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 18090
I20260809 09:27:51.687037 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:27:51.687278 9624 external_mini_cluster.cc:1428] Setting key c0812c4766e055fa715f00dfd1e8b3ed
I20260809 09:27:51.689111 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:27:51.692189 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.690803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38688 (local address 127.9.102.62:39521)
0809 09:27:51.690925 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:51.690929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:51.691075 (+ 146us) server_negotiation.cc:407] Connection header received
0809 09:27:51.691167 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:51.691169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:51.691196 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:51.691256 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:51.691848 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:51.691850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:51.691852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:51.691862 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:51.691877 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:51.691884 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:51.691952 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:51.692011 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 09:27:51.692038 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 09:27:51.694657 17551 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:27:51.697436 17551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:27:51.707952 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.705727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:46679 (local address 127.0.0.1:38968)
0809 09:27:51.705984 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:51.705999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:51.706078 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:51.706879 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:51.706883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:51.706921 (+ 38us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:51.707347 (+ 426us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:27:51.707350 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:51.707359 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:27:51.707378 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:51.707649 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:51.707658 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:51.707663 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 09:27:51.707768 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 09:27:51.707828 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":99,"threads_started":1}
I20260809 09:27:51.707983 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.706215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.9.102.3:46679)
0809 09:27:51.706486 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:51.706498 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:51.706519 (+ 21us) server_negotiation.cc:407] Connection header received
0809 09:27:51.706576 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:51.706581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:51.706747 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:51.706894 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:51.707449 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:51.707454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:51.707465 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:51.707477 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:51.707512 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:51.707549 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:51.707657 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:51.707772 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 09:27:51.707829 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":118,"threads_started":1}
I20260809 09:27:51.707952 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.705076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:41503 (local address 127.0.0.1:38826)
0809 09:27:51.705390 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:51.705435 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:51.705529 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:51.706527 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:51.706533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:51.706597 (+ 64us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:51.707130 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:27:51.707134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:51.707154 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:27:51.707215 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:51.707805 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:51.707807 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:51.707810 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 09:27:51.707852 (+ 42us) client_negotiation.cc:240] Negotiation successful
0809 09:27:51.707874 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260809 09:27:51.708307 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.705175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.9.102.2:41503)
0809 09:27:51.706133 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:51.706146 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:51.706173 (+ 27us) server_negotiation.cc:407] Connection header received
0809 09:27:51.706224 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:51.706229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:51.706376 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:51.706529 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:51.707632 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:51.707639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:51.707648 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:51.707658 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:51.707691 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:51.707715 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:51.708028 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:51.708073 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 09:27:51.708158 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":133,"threads_started":1}
I20260809 09:27:51.709601 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.705758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:36599 (local address 127.0.0.1:48824)
0809 09:27:51.706030 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:51.706044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:51.706123 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:51.708183 (+ 2060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:51.708186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:51.708199 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:51.708697 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:27:51.708699 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:51.708710 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:27:51.708736 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:51.709348 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:51.709351 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:51.709354 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 09:27:51.709461 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 09:27:51.709494 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":108,"threads_started":1}
I20260809 09:27:51.710120 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.706562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.9.102.1:36599)
0809 09:27:51.707635 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:51.707645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:51.707666 (+ 21us) server_negotiation.cc:407] Connection header received
0809 09:27:51.707712 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:51.707716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:51.707852 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:51.707962 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:51.709179 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:51.709183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:51.709192 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:51.709201 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:51.709232 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:51.709252 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:51.709354 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:51.709898 (+ 544us) server_negotiation.cc:299] Negotiation successful
0809 09:27:51.709960 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":97,"threads_started":1}
I20260809 09:27:51.710755 18188 tablet_service.cc:1467] Processing CreateTablet for tablet c60655e3901d4a39947a2c1ed615dec1 (DEFAULT_TABLE table=test-table [id=4b799a2265d74c75b41b3a98887fdbf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:51.711014 17981 tablet_service.cc:1467] Processing CreateTablet for tablet c60655e3901d4a39947a2c1ed615dec1 (DEFAULT_TABLE table=test-table [id=4b799a2265d74c75b41b3a98887fdbf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:51.711727 17981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60655e3901d4a39947a2c1ed615dec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:51.711755 18188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60655e3901d4a39947a2c1ed615dec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:51.712667 17760 tablet_service.cc:1467] Processing CreateTablet for tablet c60655e3901d4a39947a2c1ed615dec1 (DEFAULT_TABLE table=test-table [id=4b799a2265d74c75b41b3a98887fdbf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:51.713498 17760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c60655e3901d4a39947a2c1ed615dec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:51.718102 18328 tablet_bootstrap.cc:492] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f: Bootstrap starting.
I20260809 09:27:51.719769 18332 tablet_bootstrap.cc:492] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc: Bootstrap starting.
I20260809 09:27:51.719861 18328 tablet_bootstrap.cc:654] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:51.720311 18330 tablet_bootstrap.cc:492] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268: Bootstrap starting.
I20260809 09:27:51.720670 18328 log.cc:824] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:51.721776 18332 tablet_bootstrap.cc:654] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:51.722199 18330 tablet_bootstrap.cc:654] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:51.722566 18332 log.cc:824] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:51.722810 18328 tablet_bootstrap.cc:492] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f: No bootstrap required, opened a new log
I20260809 09:27:51.722913 18330 log.cc:824] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:51.722999 18328 ts_tablet_manager.cc:1397] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260809 09:27:51.724406 18332 tablet_bootstrap.cc:492] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc: No bootstrap required, opened a new log
I20260809 09:27:51.724576 18330 tablet_bootstrap.cc:492] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268: No bootstrap required, opened a new log
I20260809 09:27:51.724583 18332 ts_tablet_manager.cc:1397] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260809 09:27:51.724773 18330 ts_tablet_manager.cc:1397] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:27:51.727265 18328 raft_consensus.cc:348] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2b7a619d0d4125ac652d18189f289f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41503 } } peers { permanent_uuid: "d91445dbe6ef43a6912df6cd05382268" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46679 } } peers { permanent_uuid: "8f1884cfac75438f9d69555de30eddbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36599 } }
I20260809 09:27:51.727533 18328 raft_consensus.cc:374] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:51.727586 18328 raft_consensus.cc:729] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2b7a619d0d4125ac652d18189f289f, State: Initialized, Role: FOLLOWER
I20260809 09:27:51.728009 18328 consensus_queue.cc:260] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2b7a619d0d4125ac652d18189f289f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41503 } } peers { permanent_uuid: "d91445dbe6ef43a6912df6cd05382268" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46679 } } peers { permanent_uuid: "8f1884cfac75438f9d69555de30eddbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36599 } }
I20260809 09:27:51.727907 18330 raft_consensus.cc:348] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2b7a619d0d4125ac652d18189f289f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41503 } } peers { permanent_uuid: "d91445dbe6ef43a6912df6cd05382268" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46679 } } peers { permanent_uuid: "8f1884cfac75438f9d69555de30eddbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36599 } }
I20260809 09:27:51.728179 18330 raft_consensus.cc:374] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:51.728231 18330 raft_consensus.cc:729] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d91445dbe6ef43a6912df6cd05382268, State: Initialized, Role: FOLLOWER
I20260809 09:27:51.728646 18330 consensus_queue.cc:260] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2b7a619d0d4125ac652d18189f289f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41503 } } peers { permanent_uuid: "d91445dbe6ef43a6912df6cd05382268" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46679 } } peers { permanent_uuid: "8f1884cfac75438f9d69555de30eddbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36599 } }
I20260809 09:27:51.728775 18332 raft_consensus.cc:348] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2b7a619d0d4125ac652d18189f289f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41503 } } peers { permanent_uuid: "d91445dbe6ef43a6912df6cd05382268" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46679 } } peers { permanent_uuid: "8f1884cfac75438f9d69555de30eddbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36599 } }
I20260809 09:27:51.728991 18328 ts_tablet_manager.cc:1428] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 09:27:51.729056 18332 raft_consensus.cc:374] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:51.729120 18332 raft_consensus.cc:729] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1884cfac75438f9d69555de30eddbc, State: Initialized, Role: FOLLOWER
I20260809 09:27:51.729331 18086 heartbeater.cc:503] Master 127.9.102.62:39521 was elected leader, sending a full tablet report...
I20260809 09:27:51.729395 18298 heartbeater.cc:503] Master 127.9.102.62:39521 was elected leader, sending a full tablet report...
I20260809 09:27:51.729537 18332 consensus_queue.cc:260] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2b7a619d0d4125ac652d18189f289f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41503 } } peers { permanent_uuid: "d91445dbe6ef43a6912df6cd05382268" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46679 } } peers { permanent_uuid: "8f1884cfac75438f9d69555de30eddbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36599 } }
I20260809 09:27:51.729607 18330 ts_tablet_manager.cc:1428] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:27:51.730406 18332 ts_tablet_manager.cc:1428] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260809 09:27:51.730705 17872 heartbeater.cc:503] Master 127.9.102.62:39521 was elected leader, sending a full tablet report...
W20260809 09:27:51.765513 18088 tablet.cc:2369] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:27:51.843405 18336 raft_consensus.cc:482] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:27:51.843611 18336 raft_consensus.cc:504] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2b7a619d0d4125ac652d18189f289f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41503 } } peers { permanent_uuid: "d91445dbe6ef43a6912df6cd05382268" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46679 } } peers { permanent_uuid: "8f1884cfac75438f9d69555de30eddbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36599 } }
I20260809 09:27:51.844647 18336 leader_election.cc:288] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d91445dbe6ef43a6912df6cd05382268 (127.9.102.3:46679), 8f1884cfac75438f9d69555de30eddbc (127.9.102.1:36599)
I20260809 09:27:51.846608 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.844955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:39475 (local address 127.9.102.3:46679)
0809 09:27:51.845079 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:51.845083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:51.845335 (+ 252us) server_negotiation.cc:407] Connection header received
0809 09:27:51.845454 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:51.845457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:51.845507 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:51.845589 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:51.846178 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:51.846181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:51.846182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:51.846210 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:51.846228 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:51.846235 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:51.846316 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:51.846431 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 09:27:51.846471 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 09:27:51.846612 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.844868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:36599 (local address 127.9.102.2:44333)
0809 09:27:51.844963 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:51.844979 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:51.845046 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:51.845341 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:51.845344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:51.845357 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:51.845836 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:27:51.845839 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:51.845853 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:27:51.845887 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:51.846351 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:51.846353 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:51.846356 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 09:27:51.846448 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 09:27:51.846483 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260809 09:27:51.846612 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.845058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:44333 (local address 127.9.102.1:36599)
0809 09:27:51.845169 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:51.845171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:51.845181 (+ 10us) server_negotiation.cc:407] Connection header received
0809 09:27:51.845213 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:51.845214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:51.845241 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:51.845322 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:51.846022 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:51.846026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:51.846029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:51.846041 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:51.846061 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:51.846072 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:51.846158 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:51.846457 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 09:27:51.846494 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 09:27:51.846647 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.844868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:46679 (local address 127.9.102.2:39475)
0809 09:27:51.845238 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:51.845249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:51.845358 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:51.845611 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:51.845613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:51.845627 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:51.846016 (+ 389us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:27:51.846017 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:51.846035 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:27:51.846060 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:51.846324 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:51.846326 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:51.846329 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 09:27:51.846422 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 09:27:51.846459 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":34,"thread_start_us":122,"threads_started":1}
I20260809 09:27:51.847291 17790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60655e3901d4a39947a2c1ed615dec1" candidate_uuid: "8d2b7a619d0d4125ac652d18189f289f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1884cfac75438f9d69555de30eddbc" is_pre_election: true
I20260809 09:27:51.847342 18219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60655e3901d4a39947a2c1ed615dec1" candidate_uuid: "8d2b7a619d0d4125ac652d18189f289f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91445dbe6ef43a6912df6cd05382268" is_pre_election: true
I20260809 09:27:51.847549 17790 raft_consensus.cc:2454] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2b7a619d0d4125ac652d18189f289f in term 0.
I20260809 09:27:51.847604 18219 raft_consensus.cc:2454] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d2b7a619d0d4125ac652d18189f289f in term 0.
I20260809 09:27:51.848084 17905 leader_election.cc:302] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d2b7a619d0d4125ac652d18189f289f, 8f1884cfac75438f9d69555de30eddbc; no voters:
I20260809 09:27:51.848388 18336 raft_consensus.cc:2781] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:27:51.848459 18336 raft_consensus.cc:482] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:27:51.848510 18336 raft_consensus.cc:3037] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:51.849517 18336 raft_consensus.cc:504] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2b7a619d0d4125ac652d18189f289f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41503 } } peers { permanent_uuid: "d91445dbe6ef43a6912df6cd05382268" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46679 } } peers { permanent_uuid: "8f1884cfac75438f9d69555de30eddbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36599 } }
I20260809 09:27:51.849833 18336 leader_election.cc:288] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [CANDIDATE]: Term 1 election: Requested vote from peers d91445dbe6ef43a6912df6cd05382268 (127.9.102.3:46679), 8f1884cfac75438f9d69555de30eddbc (127.9.102.1:36599)
I20260809 09:27:51.850170 18219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60655e3901d4a39947a2c1ed615dec1" candidate_uuid: "8d2b7a619d0d4125ac652d18189f289f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91445dbe6ef43a6912df6cd05382268"
I20260809 09:27:51.850170 17790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c60655e3901d4a39947a2c1ed615dec1" candidate_uuid: "8d2b7a619d0d4125ac652d18189f289f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1884cfac75438f9d69555de30eddbc"
I20260809 09:27:51.850291 17790 raft_consensus.cc:3037] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:51.850283 18219 raft_consensus.cc:3037] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268 [term 0 FOLLOWER]: Advancing to term 1
W20260809 09:27:51.851125 17873 tablet.cc:2369] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:27:51.851444 18219 raft_consensus.cc:2454] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2b7a619d0d4125ac652d18189f289f in term 1.
I20260809 09:27:51.851444 17790 raft_consensus.cc:2454] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d2b7a619d0d4125ac652d18189f289f in term 1.
I20260809 09:27:51.851709 17902 leader_election.cc:302] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d2b7a619d0d4125ac652d18189f289f, d91445dbe6ef43a6912df6cd05382268; no voters:
I20260809 09:27:51.851883 18336 raft_consensus.cc:2781] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:51.852125 18336 raft_consensus.cc:686] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 1 LEADER]: Becoming Leader. State: Replica: 8d2b7a619d0d4125ac652d18189f289f, State: Running, Role: LEADER
I20260809 09:27:51.852437 18336 consensus_queue.cc:237] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2b7a619d0d4125ac652d18189f289f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41503 } } peers { permanent_uuid: "d91445dbe6ef43a6912df6cd05382268" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46679 } } peers { permanent_uuid: "8f1884cfac75438f9d69555de30eddbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36599 } }
I20260809 09:27:51.855506 17551 catalog_manager.cc:5495] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d2b7a619d0d4125ac652d18189f289f (127.9.102.2). New cstate: current_term: 1 leader_uuid: "8d2b7a619d0d4125ac652d18189f289f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d2b7a619d0d4125ac652d18189f289f" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 41503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d91445dbe6ef43a6912df6cd05382268" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f1884cfac75438f9d69555de30eddbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36599 } health_report { overall_health: UNKNOWN } } }
I20260809 09:27:51.894801 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.893413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.9.102.2:41503)
0809 09:27:51.893571 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:51.893576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:51.893588 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:27:51.893753 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:51.893755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:51.893805 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:51.893888 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:51.894449 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:51.894452 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:51.894454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:51.894464 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:51.894482 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:51.894492 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:51.894582 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:51.894657 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 09:27:51.894690 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 09:27:51.899982 18219 raft_consensus.cc:1261] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268 [term 1 FOLLOWER]: Refusing update from remote peer 8d2b7a619d0d4125ac652d18189f289f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:51.899982 17790 raft_consensus.cc:1261] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc [term 1 FOLLOWER]: Refusing update from remote peer 8d2b7a619d0d4125ac652d18189f289f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:51.900642 18336 consensus_queue.cc:1035] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1884cfac75438f9d69555de30eddbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 36599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:51.900825 18342 consensus_queue.cc:1035] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d91445dbe6ef43a6912df6cd05382268" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:51.907215 18347 mvcc.cc:205] Tried to move back new op lower bound from 7316552384095514624 to 7316552383910506496. Current Snapshot: MvccSnapshot[applied={T|T < 7316552384095514624}]
I20260809 09:27:51.910732 18348 mvcc.cc:205] Tried to move back new op lower bound from 7316552384095514624 to 7316552383910506496. Current Snapshot: MvccSnapshot[applied={T|T < 7316552384095514624}]
I20260809 09:27:51.914819 17551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:51.914951 17551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:51.916918 17551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 968e31debd2243df8c8d3ac3f8ef9399: Sending DeleteTablet for 3 replicas of tablet c60655e3901d4a39947a2c1ed615dec1
I20260809 09:27:51.917578 18188 tablet_service.cc:1514] Processing DeleteTablet for tablet c60655e3901d4a39947a2c1ed615dec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:51 UTC) from {username='slave'} at 127.0.0.1:38968
I20260809 09:27:51.917579 17760 tablet_service.cc:1514] Processing DeleteTablet for tablet c60655e3901d4a39947a2c1ed615dec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:51 UTC) from {username='slave'} at 127.0.0.1:48824
I20260809 09:27:51.917999 18362 tablet_replica.cc:330] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc: stopping tablet replica
I20260809 09:27:51.918009 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 17657
I20260809 09:27:51.918256 18362 raft_consensus.cc:2229] T c60655e3901d4a39947a2c1ed615dec1 P 8f1884cfac75438f9d69555de30eddbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:51.918634 18360 tablet_replica.cc:330] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268: stopping tablet replica
I20260809 09:27:51.918726 17981 tablet_service.cc:1514] Processing DeleteTablet for tablet c60655e3901d4a39947a2c1ed615dec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:51 UTC) from {username='slave'} at 127.0.0.1:38826
I20260809 09:27:51.918905 18360 raft_consensus.cc:2229] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:51.919148 18360 raft_consensus.cc:2258] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:51.924052 18360 ts_tablet_manager.cc:1905] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:51.926139 18360 ts_tablet_manager.cc:1918] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:27:51.926251 18360 log.cc:1196] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/c60655e3901d4a39947a2c1ed615dec1
I20260809 09:27:51.926528 18360 ts_tablet_manager.cc:1939] T c60655e3901d4a39947a2c1ed615dec1 P d91445dbe6ef43a6912df6cd05382268: Deleting consensus metadata
I20260809 09:27:51.927508 18363 tablet_replica.cc:330] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f: stopping tablet replica
I20260809 09:27:51.927493 17528 catalog_manager.cc:4841] TS d91445dbe6ef43a6912df6cd05382268 (127.9.102.3:46679): tablet c60655e3901d4a39947a2c1ed615dec1 (table test-table [id=4b799a2265d74c75b41b3a98887fdbf7]) successfully deleted
I20260809 09:27:51.927752 18363 raft_consensus.cc:2229] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:27:51.928035 18363 raft_consensus.cc:2258] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:51.929564 18363 ts_tablet_manager.cc:1905] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:51.931403 18363 ts_tablet_manager.cc:1918] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:27:51.931509 18363 log.cc:1196] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/c60655e3901d4a39947a2c1ed615dec1
I20260809 09:27:51.931874 18363 ts_tablet_manager.cc:1939] T c60655e3901d4a39947a2c1ed615dec1 P 8d2b7a619d0d4125ac652d18189f289f: Deleting consensus metadata
I20260809 09:27:51.933192 17534 catalog_manager.cc:4841] TS 8d2b7a619d0d4125ac652d18189f289f (127.9.102.2:41503): tablet c60655e3901d4a39947a2c1ed615dec1 (table test-table [id=4b799a2265d74c75b41b3a98887fdbf7]) successfully deleted
W20260809 09:27:51.940665 17534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:36599 (error 108)
I20260809 09:27:51.940835 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 17876
I20260809 09:27:51.942584 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:51.941495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:36599 (local address 127.0.0.1:48826)
0809 09:27:51.942403 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:51.942488 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:36599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":802}
W20260809 09:27:51.942760 17534 catalog_manager.cc:4584] TS 8f1884cfac75438f9d69555de30eddbc (127.9.102.1:36599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c60655e3901d4a39947a2c1ed615dec1: Network error: Client connection negotiation failed: client connection to 127.9.102.1:36599: connect: Connection refused (error 111)
I20260809 09:27:51.950603 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 18090
I20260809 09:27:51.963352 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 17498
2026-08-09T09:27:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1029 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 09:27:51.989085 9624 mini_oidc.cc:152] Starting JWKS server
I20260809 09:27:51.990717 9624 webserver.cc:460] Webserver started at https://127.0.0.1:45573/ using document root <none> and password file <none>
I20260809 09:27:51.990902 9624 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 09:27:51.991176 9624 webserver.cc:460] Webserver started at http://127.0.0.1:39221/ using document root <none> and password file <none>
2026-08-09T09:27:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:51Z Disabled control of system clock
I20260809 09:27:52.000788 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:42315
--webserver_interface=127.9.102.62
--webserver_port=0
--builtin_ntp_servers=127.9.102.20:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:42315
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45573/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/testchainca.cert with env {}
W20260809 09:27:52.103119 18371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:52.103322 18371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:52.103397 18371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:52.107077 18371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:52.107121 18371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:52.107134 18371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:52.107182 18371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:52.107205 18371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 09:27:52.107232 18371 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 09:27:52.107265 18371 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45573/jwks.json
I20260809 09:27:52.111552 18371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:42315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:42315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45573/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:52.112345 18371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:52.113183 18371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:52.120376 18385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:52.120471 18382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:52.120477 18381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:52.120615 18371 server_base.cc:1034] running on GCE node
I20260809 09:27:52.121205 18371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:52.122145 18371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:52.123325 18371 hybrid_clock.cc:648] HybridClock initialized: now 1786267672123307 us; error 40 us; skew 500 ppm
I20260809 09:27:52.124925 18371 webserver.cc:460] Webserver started at http://127.9.102.62:36191/ using document root <none> and password file <none>
I20260809 09:27:52.125494 18371 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:52.125581 18371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:52.125787 18371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:52.127429 18371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "02591b2e997b4695b124fb9986193d81"
format_stamp: "Formatted at 2026-08-09 09:27:52 on dist-test-slave-7vjv"
server_key: "8148c43496feb0518160fc5121b15af8"
server_key_iv: "5f3446a5a287eb102c0cbf9b9391e370"
server_key_version: "encryptionkey@0"
I20260809 09:27:52.127938 18371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "02591b2e997b4695b124fb9986193d81"
format_stamp: "Formatted at 2026-08-09 09:27:52 on dist-test-slave-7vjv"
server_key: "8148c43496feb0518160fc5121b15af8"
server_key_iv: "5f3446a5a287eb102c0cbf9b9391e370"
server_key_version: "encryptionkey@0"
I20260809 09:27:52.131163 18371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260809 09:27:52.133311 18396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:52.134344 18371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:52.134482 18371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "02591b2e997b4695b124fb9986193d81"
format_stamp: "Formatted at 2026-08-09 09:27:52 on dist-test-slave-7vjv"
server_key: "8148c43496feb0518160fc5121b15af8"
server_key_iv: "5f3446a5a287eb102c0cbf9b9391e370"
server_key_version: "encryptionkey@0"
I20260809 09:27:52.134590 18371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:52.148422 18371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:52.148985 18371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:52.149189 18371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:52.155690 18371 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:42315
I20260809 09:27:52.155699 18484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:42315 every 8 connection(s)
I20260809 09:27:52.156675 18371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:52.159215 18485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:52.163865 18485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81: Bootstrap starting.
I20260809 09:27:52.165781 18485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:52.166026 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 18371
I20260809 09:27:52.166146 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:52.166357 9624 external_mini_cluster.cc:1428] Setting key ab62ee1ebcd49a7bab4ad67b0b9b70d2
I20260809 09:27:52.166550 18485 log.cc:824] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:52.168418 18485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81: No bootstrap required, opened a new log
I20260809 09:27:52.171396 18485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02591b2e997b4695b124fb9986193d81" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42315 } }
I20260809 09:27:52.171576 18485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:52.171631 18485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02591b2e997b4695b124fb9986193d81, State: Initialized, Role: FOLLOWER
I20260809 09:27:52.172055 18485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02591b2e997b4695b124fb9986193d81" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42315 } }
I20260809 09:27:52.172245 18485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:52.172345 18485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:52.172483 18485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:52.173286 18485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02591b2e997b4695b124fb9986193d81" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42315 } }
I20260809 09:27:52.173617 18485 leader_election.cc:302] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02591b2e997b4695b124fb9986193d81; no voters:
I20260809 09:27:52.173957 18485 leader_election.cc:288] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:52.173991 18493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:52.174170 18493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [term 1 LEADER]: Becoming Leader. State: Replica: 02591b2e997b4695b124fb9986193d81, State: Running, Role: LEADER
I20260809 09:27:52.174315 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:52.167268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.9.102.62:42315)
0809 09:27:52.167786 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:52.167798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:52.167825 (+ 27us) server_negotiation.cc:407] Connection header received
0809 09:27:52.168421 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:52.168438 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:52.168768 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:52.169056 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:52.169405 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.170219 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.171080 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.171517 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.171687 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:52.172608 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:52.172631 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:52.172648 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:52.172996 (+ 348us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:52.173047 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:52.173066 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:52.173173 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:52.173283 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 09:27:52.173546 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":116,"threads_started":1}
I20260809 09:27:52.174410 18493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02591b2e997b4695b124fb9986193d81" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42315 } }
I20260809 09:27:52.175022 18485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:52.175535 18495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02591b2e997b4695b124fb9986193d81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02591b2e997b4695b124fb9986193d81" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42315 } } }
I20260809 09:27:52.175639 18495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:52.176298 18497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02591b2e997b4695b124fb9986193d81. Latest consensus state: current_term: 1 leader_uuid: "02591b2e997b4695b124fb9986193d81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02591b2e997b4695b124fb9986193d81" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42315 } } }
I20260809 09:27:52.176373 18497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:52.176925 18500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:52.177183 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:27:52.179677 18500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:52.184481 18500 catalog_manager.cc:1347] Generated new cluster ID: 73aa2795d92b421eb38e71323920f563
I20260809 09:27:52.184553 18500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:52.215886 18500 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:52.216804 18500 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:52.227744 18500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02591b2e997b4695b124fb9986193d81: Generated new TSK 0
I20260809 09:27:52.228379 18500 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 09:27:52.257943 18536 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 09:27:52.258076 18536 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 09:27:52.258467 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:52.236959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.9.102.62:42315)
0809 09:27:52.237078 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:52.237082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:52.237296 (+ 214us) server_negotiation.cc:407] Connection header received
0809 09:27:52.237465 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:52.237467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:52.237506 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:52.237575 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0809 09:27:52.237853 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.238316 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.239478 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.239692 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.239770 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:52.240163 (+ 393us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 09:27:52.258067 (+ 17904us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45573/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 09:27:52.258331 (+ 264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49000: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45573/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 09:27:52.258543 18490 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49000: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45573/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 09:27:52.259565 18535 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:27:52.236910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:42315 (local address 127.0.0.1:49000)
0809 09:27:52.237218 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:52.237231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:52.237340 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:52.237612 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:52.237615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:52.237625 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0809 09:27:52.237736 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:52.237742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.238429 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.238433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:52.239374 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:52.239384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.239805 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.239809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:52.239904 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:52.239955 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 09:27:52.258298 (+ 18343us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45573/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 09:27:52.258502 (+ 204us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.102.62:42315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45573/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.102.62:42315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45573/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 09:27:52.260306 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 18371
2026-08-09T09:27:52Z chronyd exiting
I20260809 09:27:52.304057 9624 test_util.cc:175] -----------------------------------------------
I20260809 09:27:52.304138 9624 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786267646682796-9624-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (316 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 09:27:52.305148 9624 mini_oidc.cc:152] Starting JWKS server
I20260809 09:27:52.305896 9624 webserver.cc:460] Webserver started at https://127.0.0.1:46123/ using document root <none> and password file <none>
I20260809 09:27:52.306005 9624 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 09:27:52.306265 9624 webserver.cc:460] Webserver started at http://127.0.0.1:40153/ using document root <none> and password file <none>
2026-08-09T09:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:52Z Disabled control of system clock
I20260809 09:27:52.314968 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:43811
--webserver_interface=127.9.102.62
--webserver_port=0
--builtin_ntp_servers=127.9.102.20:35929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:43811
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46123/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/testchainca.cert with env {}
W20260809 09:27:52.412055 18547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:52.412326 18547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:52.412393 18547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:52.415640 18547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:52.415709 18547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:52.415735 18547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:52.415764 18547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:52.415793 18547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 09:27:52.415823 18547 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 09:27:52.415849 18547 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46123/jwks.json
I20260809 09:27:52.420208 18547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:43811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:43811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46123/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:52.421012 18547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:52.421881 18547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:52.428993 18563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:52.428988 18556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:52.429069 18557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:52.429277 18547 server_base.cc:1034] running on GCE node
I20260809 09:27:52.429708 18547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:52.430616 18547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:52.431767 18547 hybrid_clock.cc:648] HybridClock initialized: now 1786267672431751 us; error 28 us; skew 500 ppm
I20260809 09:27:52.433364 18547 webserver.cc:460] Webserver started at http://127.9.102.62:42489/ using document root <none> and password file <none>
I20260809 09:27:52.433830 18547 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:52.433884 18547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:52.434093 18547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:52.435859 18547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "7709958ea3974573be78068697016d9d"
format_stamp: "Formatted at 2026-08-09 09:27:52 on dist-test-slave-7vjv"
server_key: "c85ef8c9ccb2e9eeaada33236398f91f"
server_key_iv: "a217c2b7d5bed0741ee6d62d3d23904c"
server_key_version: "encryptionkey@0"
I20260809 09:27:52.436450 18547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "7709958ea3974573be78068697016d9d"
format_stamp: "Formatted at 2026-08-09 09:27:52 on dist-test-slave-7vjv"
server_key: "c85ef8c9ccb2e9eeaada33236398f91f"
server_key_iv: "a217c2b7d5bed0741ee6d62d3d23904c"
server_key_version: "encryptionkey@0"
I20260809 09:27:52.439709 18547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 09:27:52.441941 18574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:52.442993 18547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:52.443128 18547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "7709958ea3974573be78068697016d9d"
format_stamp: "Formatted at 2026-08-09 09:27:52 on dist-test-slave-7vjv"
server_key: "c85ef8c9ccb2e9eeaada33236398f91f"
server_key_iv: "a217c2b7d5bed0741ee6d62d3d23904c"
server_key_version: "encryptionkey@0"
I20260809 09:27:52.443218 18547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:52.464011 18547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:52.464560 18547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:52.464725 18547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:52.470786 18547 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:43811
I20260809 09:27:52.470795 18674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:43811 every 8 connection(s)
I20260809 09:27:52.471556 18547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:52.474180 18676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:52.478999 18676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d: Bootstrap starting.
I20260809 09:27:52.479727 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 18547
I20260809 09:27:52.479838 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:52.480146 9624 external_mini_cluster.cc:1428] Setting key e274d2e3e698c3c480f0190949b2d335
I20260809 09:27:52.481678 18676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:52.482393 18676 log.cc:824] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:52.484048 18676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d: No bootstrap required, opened a new log
I20260809 09:27:52.487217 18676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709958ea3974573be78068697016d9d" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43811 } }
I20260809 09:27:52.487442 18676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:52.487522 18676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7709958ea3974573be78068697016d9d, State: Initialized, Role: FOLLOWER
I20260809 09:27:52.488000 18676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709958ea3974573be78068697016d9d" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43811 } }
I20260809 09:27:52.488198 18676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:52.488269 18676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:52.488386 18676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:52.488782 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:52.481155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.9.102.62:43811)
0809 09:27:52.481743 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:52.481758 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:52.481834 (+ 76us) server_negotiation.cc:407] Connection header received
0809 09:27:52.482421 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:52.482435 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:52.482902 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:52.483254 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:52.483631 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.484581 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.485513 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.485978 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.486162 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:52.486991 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:52.486996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:52.487016 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:52.487358 (+ 342us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:52.487406 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:52.487424 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:52.487537 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:52.487745 (+ 208us) server_negotiation.cc:299] Negotiation successful
0809 09:27:52.487973 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
I20260809 09:27:52.489246 18676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709958ea3974573be78068697016d9d" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43811 } }
I20260809 09:27:52.489569 18676 leader_election.cc:302] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7709958ea3974573be78068697016d9d; no voters:
I20260809 09:27:52.489880 18676 leader_election.cc:288] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:52.489948 18686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:52.490151 18686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [term 1 LEADER]: Becoming Leader. State: Replica: 7709958ea3974573be78068697016d9d, State: Running, Role: LEADER
I20260809 09:27:52.490442 18686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709958ea3974573be78068697016d9d" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43811 } }
I20260809 09:27:52.491070 18676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:52.491645 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:27:52.491873 18687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7709958ea3974573be78068697016d9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709958ea3974573be78068697016d9d" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43811 } } }
I20260809 09:27:52.492005 18687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:52.492326 18688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7709958ea3974573be78068697016d9d. Latest consensus state: current_term: 1 leader_uuid: "7709958ea3974573be78068697016d9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7709958ea3974573be78068697016d9d" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43811 } } }
I20260809 09:27:52.492400 18688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d [sys.catalog]: This master's current role is: LEADER
W20260809 09:27:52.495430 18708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:27:52.495520 18708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:27:52.495585 18699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:52.496280 18699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:52.501394 18699 catalog_manager.cc:1347] Generated new cluster ID: 3ce667d74fb54ff390bfbc5c45fb3d70
I20260809 09:27:52.501451 18699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:52.512249 18699 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:52.513000 18699 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:52.516052 18699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7709958ea3974573be78068697016d9d: Generated new TSK 0
I20260809 09:27:52.516706 18699 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 09:27:52.551259 18718 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 09:27:52.551331 18718 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 09:27:52.551767 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:52.531576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.9.102.62:43811)
0809 09:27:52.531692 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:52.531696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:52.531879 (+ 183us) server_negotiation.cc:407] Connection header received
0809 09:27:52.532008 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:52.532010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:52.532050 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:52.532182 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0809 09:27:52.532409 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.532844 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.533906 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.534084 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.534145 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:52.534429 (+ 284us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 09:27:52.551488 (+ 17059us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 09:27:52.551658 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46710: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":48}
W20260809 09:27:52.551839 18682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46710: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260809 09:27:52.552014 18717 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:27:52.531517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:43811 (local address 127.0.0.1:46710)
0809 09:27:52.531795 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:52.531809 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:52.531909 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:52.532193 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:52.532196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:52.532203 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0809 09:27:52.532310 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:52.532316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.532945 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.532948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:52.533798 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:52.533807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.534183 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.534187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:52.534269 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:52.534295 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 09:27:52.551655 (+ 17360us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 09:27:52.551829 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.102.62:43811: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46123/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":104,"threads_started":1}
I20260809 09:27:52.552644 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 18547
2026-08-09T09:27:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (295 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 09:27:52.600716 9624 mini_oidc.cc:152] Starting JWKS server
I20260809 09:27:52.601521 9624 webserver.cc:460] Webserver started at https://127.0.0.1:34359/ using document root <none> and password file <none>
I20260809 09:27:52.601616 9624 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 09:27:52.601842 9624 webserver.cc:460] Webserver started at http://127.0.0.1:35629/ using document root <none> and password file <none>
2026-08-09T09:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:52Z Disabled control of system clock
I20260809 09:27:52.610276 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:44873
--webserver_interface=127.9.102.62
--webserver_port=0
--builtin_ntp_servers=127.9.102.20:34841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:44873
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34359/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 09:27:52.707607 18729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:52.707834 18729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:52.707919 18729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:52.711588 18729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:52.711663 18729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:52.711689 18729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:52.711723 18729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:52.711755 18729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 09:27:52.711788 18729 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 09:27:52.711818 18729 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34359/jwks.json
I20260809 09:27:52.715811 18729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:44873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:44873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34359/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:52.716642 18729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:52.717713 18729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:52.725759 18737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:52.725816 18729 server_base.cc:1034] running on GCE node
W20260809 09:27:52.725827 18738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:52.725776 18743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:52.726416 18729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:52.727279 18729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:52.728444 18729 hybrid_clock.cc:648] HybridClock initialized: now 1786267672728426 us; error 34 us; skew 500 ppm
I20260809 09:27:52.729950 18729 webserver.cc:460] Webserver started at http://127.9.102.62:45643/ using document root <none> and password file <none>
I20260809 09:27:52.730433 18729 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:52.730516 18729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:52.730720 18729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:52.732412 18729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "74a1bb433b024302b8469fdda296a16d"
format_stamp: "Formatted at 2026-08-09 09:27:52 on dist-test-slave-7vjv"
server_key: "fe366c122816b7b0a64dcd2fbd9c9c87"
server_key_iv: "77b2b9863181bcc051fb31f5348e65e0"
server_key_version: "encryptionkey@0"
I20260809 09:27:52.732900 18729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "74a1bb433b024302b8469fdda296a16d"
format_stamp: "Formatted at 2026-08-09 09:27:52 on dist-test-slave-7vjv"
server_key: "fe366c122816b7b0a64dcd2fbd9c9c87"
server_key_iv: "77b2b9863181bcc051fb31f5348e65e0"
server_key_version: "encryptionkey@0"
I20260809 09:27:52.735951 18729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:27:52.737982 18752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:52.738915 18729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:52.739042 18729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "74a1bb433b024302b8469fdda296a16d"
format_stamp: "Formatted at 2026-08-09 09:27:52 on dist-test-slave-7vjv"
server_key: "fe366c122816b7b0a64dcd2fbd9c9c87"
server_key_iv: "77b2b9863181bcc051fb31f5348e65e0"
server_key_version: "encryptionkey@0"
I20260809 09:27:52.739140 18729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:52.767946 18729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:52.768592 18729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:52.768790 18729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:52.775259 18729 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:44873
I20260809 09:27:52.775266 18837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:44873 every 8 connection(s)
I20260809 09:27:52.776065 18729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:52.778741 18842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:52.783423 18842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d: Bootstrap starting.
I20260809 09:27:52.785539 18842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:52.785883 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 18729
I20260809 09:27:52.786001 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:52.786275 9624 external_mini_cluster.cc:1428] Setting key d41c4638023c9d9a8c67e70597b6b6ad
I20260809 09:27:52.786314 18842 log.cc:824] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:52.788162 18842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d: No bootstrap required, opened a new log
I20260809 09:27:52.791182 18842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a1bb433b024302b8469fdda296a16d" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44873 } }
I20260809 09:27:52.791400 18842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:52.791460 18842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a1bb433b024302b8469fdda296a16d, State: Initialized, Role: FOLLOWER
I20260809 09:27:52.791869 18842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a1bb433b024302b8469fdda296a16d" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44873 } }
I20260809 09:27:52.792007 18842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:52.792076 18842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:52.792184 18842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:52.792984 18842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a1bb433b024302b8469fdda296a16d" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44873 } }
I20260809 09:27:52.793332 18842 leader_election.cc:302] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74a1bb433b024302b8469fdda296a16d; no voters:
I20260809 09:27:52.793633 18842 leader_election.cc:288] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:52.793738 18850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:52.793951 18850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [term 1 LEADER]: Becoming Leader. State: Replica: 74a1bb433b024302b8469fdda296a16d, State: Running, Role: LEADER
I20260809 09:27:52.794255 18850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a1bb433b024302b8469fdda296a16d" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44873 } }
I20260809 09:27:52.794757 18842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:52.795172 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:52.787261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48798 (local address 127.9.102.62:44873)
0809 09:27:52.787785 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:52.787799 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:52.788007 (+ 208us) server_negotiation.cc:407] Connection header received
0809 09:27:52.788762 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:52.788777 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:52.789122 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:52.789426 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:52.789808 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.790680 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.791565 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.792206 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.792375 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:52.793103 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:52.793109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:52.793123 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:52.793550 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:52.793614 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:52.793641 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:52.793734 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:52.793961 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 09:27:52.794337 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":111,"threads_started":1}
I20260809 09:27:52.795281 18850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74a1bb433b024302b8469fdda296a16d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a1bb433b024302b8469fdda296a16d" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44873 } } }
I20260809 09:27:52.795375 18850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:52.795535 18850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74a1bb433b024302b8469fdda296a16d. Latest consensus state: current_term: 1 leader_uuid: "74a1bb433b024302b8469fdda296a16d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a1bb433b024302b8469fdda296a16d" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44873 } } }
I20260809 09:27:52.795629 18850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:52.797821 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260809 09:27:52.799667 18881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:27:52.799746 18881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:27:52.799821 18860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:52.800518 18860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:52.805599 18860 catalog_manager.cc:1347] Generated new cluster ID: 55b53b0af823481087c703486c593f7c
I20260809 09:27:52.805667 18860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:52.819466 18860 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:52.820210 18860 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:52.827986 18860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74a1bb433b024302b8469fdda296a16d: Generated new TSK 0
I20260809 09:27:52.828612 18860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:27:52.848685 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:52.844181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.9.102.62:44873)
0809 09:27:52.844302 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:52.844306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:52.844499 (+ 193us) server_negotiation.cc:407] Connection header received
0809 09:27:52.844635 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:52.844638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:52.844678 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:52.844756 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0809 09:27:52.844988 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.845445 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.846566 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.846749 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.846822 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:52.847104 (+ 282us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 09:27:52.848308 (+ 1204us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 09:27:52.848537 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 09:27:52.848745 18849 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 09:27:52.850585 18894 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:27:52.844138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:44873 (local address 127.0.0.1:48804)
0809 09:27:52.844414 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:52.844427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:52.844531 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:52.844780 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:52.844783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:52.844789 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0809 09:27:52.844888 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:52.844893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.845562 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.845566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:52.846447 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:52.846456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:52.846853 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:52.846857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:52.846934 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:52.846959 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 09:27:52.848477 (+ 1518us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 09:27:52.850413 (+ 1936us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.102.62:44873: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34359/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":1749,"thread_start_us":112,"threads_started":1}
I20260809 09:27:52.851065 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 18729
E20260809 09:27:52.871791 18895 webserver.cc:533] Webserver: sslize failed:
E20260809 09:27:52.871874 18895 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-09T09:27:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T09:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:52Z Disabled control of system clock
I20260809 09:27:52.911446 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:44373
--webserver_interface=127.9.102.62
--webserver_port=0
--builtin_ntp_servers=127.9.102.20:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:44373
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/testchainca.cert with env {}
W20260809 09:27:53.013319 18899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:53.013572 18899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:53.013672 18899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:53.017194 18899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:53.017266 18899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:53.017305 18899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:53.017359 18899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:53.017418 18899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 09:27:53.017450 18899 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 09:27:53.017478 18899 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 09:27:53.022267 18899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:44373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:44373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:53.023221 18899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:53.024243 18899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:53.031944 18911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:53.032037 18899 server_base.cc:1034] running on GCE node
W20260809 09:27:53.031934 18907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:53.031951 18906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:53.032699 18899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:53.033603 18899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:53.034786 18899 hybrid_clock.cc:648] HybridClock initialized: now 1786267673034771 us; error 30 us; skew 500 ppm
I20260809 09:27:53.036577 18899 webserver.cc:460] Webserver started at http://127.9.102.62:45647/ using document root <none> and password file <none>
I20260809 09:27:53.037072 18899 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:53.037154 18899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:53.037356 18899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:53.039113 18899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "7597f906949b4e1aaa8adbddf9a89108"
format_stamp: "Formatted at 2026-08-09 09:27:53 on dist-test-slave-7vjv"
server_key: "b822e6ab79075d3b0f11b96b6e4280af"
server_key_iv: "2b1e53d1ae32da38dcae1d60ad589107"
server_key_version: "encryptionkey@0"
I20260809 09:27:53.039636 18899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "7597f906949b4e1aaa8adbddf9a89108"
format_stamp: "Formatted at 2026-08-09 09:27:53 on dist-test-slave-7vjv"
server_key: "b822e6ab79075d3b0f11b96b6e4280af"
server_key_iv: "2b1e53d1ae32da38dcae1d60ad589107"
server_key_version: "encryptionkey@0"
I20260809 09:27:53.042865 18899 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 09:27:53.045012 18919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:53.046069 18899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:53.046228 18899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "7597f906949b4e1aaa8adbddf9a89108"
format_stamp: "Formatted at 2026-08-09 09:27:53 on dist-test-slave-7vjv"
server_key: "b822e6ab79075d3b0f11b96b6e4280af"
server_key_iv: "2b1e53d1ae32da38dcae1d60ad589107"
server_key_version: "encryptionkey@0"
I20260809 09:27:53.046325 18899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:53.054524 18899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:53.055083 18899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:53.055250 18899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:53.062448 18899 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:44373
I20260809 09:27:53.062454 19007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:44373 every 8 connection(s)
I20260809 09:27:53.063210 18899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:53.065878 19009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:53.066170 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 18899
I20260809 09:27:53.066289 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:53.066560 9624 external_mini_cluster.cc:1428] Setting key 9208cc81532d7711253b93414468aa85
I20260809 09:27:53.072050 19009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108: Bootstrap starting.
I20260809 09:27:53.074311 19009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:53.074528 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:53.067712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.9.102.62:44373)
0809 09:27:53.068283 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:53.068296 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:53.068327 (+ 31us) server_negotiation.cc:407] Connection header received
0809 09:27:53.068916 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:53.068931 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:53.069249 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:53.069490 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:53.069819 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:53.070633 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:53.071725 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:53.072164 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:53.072290 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:53.073062 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:53.073067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:53.073077 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:27:53.073385 (+ 308us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:53.073434 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:27:53.073450 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:53.073519 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:53.073679 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 09:27:53.073937 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":106,"threads_started":1}
I20260809 09:27:53.074988 19009 log.cc:824] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:53.076699 19009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108: No bootstrap required, opened a new log
I20260809 09:27:53.076884 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:27:53.079818 19009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7597f906949b4e1aaa8adbddf9a89108" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44373 } }
I20260809 09:27:53.079994 19009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:53.080025 19009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7597f906949b4e1aaa8adbddf9a89108, State: Initialized, Role: FOLLOWER
I20260809 09:27:53.080394 19009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7597f906949b4e1aaa8adbddf9a89108" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44373 } }
I20260809 09:27:53.080500 19009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:53.080547 19009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:53.080611 19009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:53.081292 19009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7597f906949b4e1aaa8adbddf9a89108" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44373 } }
I20260809 09:27:53.081554 19009 leader_election.cc:302] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7597f906949b4e1aaa8adbddf9a89108; no voters:
I20260809 09:27:53.081763 19009 leader_election.cc:288] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:53.081876 19018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:53.082116 19018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [term 1 LEADER]: Becoming Leader. State: Replica: 7597f906949b4e1aaa8adbddf9a89108, State: Running, Role: LEADER
I20260809 09:27:53.082412 19018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7597f906949b4e1aaa8adbddf9a89108" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44373 } }
I20260809 09:27:53.082892 19009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:53.083998 19020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7597f906949b4e1aaa8adbddf9a89108. Latest consensus state: current_term: 1 leader_uuid: "7597f906949b4e1aaa8adbddf9a89108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7597f906949b4e1aaa8adbddf9a89108" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44373 } } }
I20260809 09:27:53.084138 19020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:53.084445 19019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7597f906949b4e1aaa8adbddf9a89108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7597f906949b4e1aaa8adbddf9a89108" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 44373 } } }
I20260809 09:27:53.084527 19019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108 [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:53.084609 19030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:53.087020 19030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:53.091603 19030 catalog_manager.cc:1347] Generated new cluster ID: e774c77a32904c7eb9e66712654c60c1
I20260809 09:27:53.091670 19030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:53.105521 19030 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:53.106307 19030 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:53.110105 19030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7597f906949b4e1aaa8adbddf9a89108: Generated new TSK 0
I20260809 09:27:53.110625 19030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:27:53.168401 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:53.118565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50930 (local address 127.9.102.62:44373)
0809 09:27:53.118666 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:53.118669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:53.118801 (+ 132us) server_negotiation.cc:407] Connection header received
0809 09:27:53.118925 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:53.118927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:53.118964 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:53.119036 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0809 09:27:53.119263 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:53.119743 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:53.120839 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:53.121016 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:53.121086 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:53.121382 (+ 296us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 09:27:53.168084 (+ 46702us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 09:27:53.168312 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50930: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 09:27:53.168473 19014 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50930: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 09:27:53.168607 19060 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:27:53.118438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:44373 (local address 127.0.0.1:50930)
0809 09:27:53.118716 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:53.118729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:53.118827 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:53.119059 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:53.119062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:53.119069 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0809 09:27:53.119167 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:53.119173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:53.119844 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:53.119847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:53.120731 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:53.120740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:53.121129 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:53.121132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:53.121214 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:53.121239 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 09:27:53.168284 (+ 47045us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 09:27:53.168446 (+ 162us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.102.62:44373: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":93,"threads_started":1}
I20260809 09:27:53.169108 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 18899
2026-08-09T09:27:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (297 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (112 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (108 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:27:53 dist-test-slave-7vjv krb5kdc[19083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:27:53 dist-test-slave-7vjv krb5kdc[19083](info): set up 2 sockets
Aug 09 09:27:53 dist-test-slave-7vjv krb5kdc[19083](info): commencing operation
krb5kdc: starting...
W20260809 09:27:54.558130 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:27:54 dist-test-slave-7vjv krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:27:54.664870 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:36151
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:36151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:54.764437 19100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:54.764683 19100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:54.764747 19100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:54.769001 19100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:54.769076 19100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:54.769111 19100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:54.769145 19100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:54.769178 19100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:54.773655 19100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:36151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:36151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:54.774529 19100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:54.775375 19100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:54.782959 19110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:54.782958 19109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:54.783151 19100 server_base.cc:1034] running on GCE node
W20260809 09:27:54.783037 19112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:54.783758 19100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:54.784698 19100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:54.785857 19100 hybrid_clock.cc:648] HybridClock initialized: now 1786267674785831 us; error 35 us; skew 500 ppm
Aug 09 09:27:54 dist-test-slave-7vjv krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267674, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:54.790832 19100 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:27:54.791592 19100 webserver.cc:460] Webserver started at http://127.9.102.62:41781/ using document root <none> and password file <none>
I20260809 09:27:54.792080 19100 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:54.792251 19100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:54.792484 19100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:54.794217 19100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "a369aefa97e8428d972edd4204277fbd"
format_stamp: "Formatted at 2026-08-09 09:27:54 on dist-test-slave-7vjv"
server_key: "e78fd1fbaf591c3f4a5c6ca6ab805ec4"
server_key_iv: "5dc97c85247a119d91056ba6a050283c"
server_key_version: "encryptionkey@0"
I20260809 09:27:54.794734 19100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "a369aefa97e8428d972edd4204277fbd"
format_stamp: "Formatted at 2026-08-09 09:27:54 on dist-test-slave-7vjv"
server_key: "e78fd1fbaf591c3f4a5c6ca6ab805ec4"
server_key_iv: "5dc97c85247a119d91056ba6a050283c"
server_key_version: "encryptionkey@0"
I20260809 09:27:54.798131 19100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 09:27:54.800336 19122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:54.801435 19100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 09:27:54.801585 19100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "a369aefa97e8428d972edd4204277fbd"
format_stamp: "Formatted at 2026-08-09 09:27:54 on dist-test-slave-7vjv"
server_key: "e78fd1fbaf591c3f4a5c6ca6ab805ec4"
server_key_iv: "5dc97c85247a119d91056ba6a050283c"
server_key_version: "encryptionkey@0"
I20260809 09:27:54.801690 19100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:54.817723 19100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:54.820322 19100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:54.820499 19100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:54.826958 19100 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:36151
I20260809 09:27:54.826970 19208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:36151 every 8 connection(s)
I20260809 09:27:54.827757 19100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:54.830114 19209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:54.830456 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 19100
I20260809 09:27:54.830605 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:54.830860 9624 external_mini_cluster.cc:1428] Setting key cda5fbd1857336156076468c81aa74ee
I20260809 09:27:54.836846 19209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd: Bootstrap starting.
I20260809 09:27:54.838901 19209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd: Neither blocks nor log segments found. Creating new log.
Aug 09 09:27:54 dist-test-slave-7vjv krb5kdc[19083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:54.839819 19209 log.cc:824] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:54.841843 19209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd: No bootstrap required, opened a new log
I20260809 09:27:54.844659 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:54.832138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.9.102.62:36151)
0809 09:27:54.832516 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:54.832541 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:54.832579 (+ 38us) server_negotiation.cc:407] Connection header received
0809 09:27:54.833313 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:54.833337 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:54.833731 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:54.834074 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:54.834752 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:54.836228 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:54.837055 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:54.837488 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:54.837654 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:54.840382 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:54.840392 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:54.840410 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:54.840437 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:54.842095 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:54.842554 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:54.842558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:54.842562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:54.842672 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:54.842941 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:54.842942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:54.842943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:54.843281 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:54.843422 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:54.843716 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 09:27:54.843946 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260809 09:27:54.845607 19209 raft_consensus.cc:348] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a369aefa97e8428d972edd4204277fbd" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 36151 } }
I20260809 09:27:54.845822 19209 raft_consensus.cc:374] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:54.845899 19209 raft_consensus.cc:729] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a369aefa97e8428d972edd4204277fbd, State: Initialized, Role: FOLLOWER
I20260809 09:27:54.846340 19209 consensus_queue.cc:260] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a369aefa97e8428d972edd4204277fbd" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 36151 } }
I20260809 09:27:54.846503 19209 raft_consensus.cc:388] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:54.846593 19209 raft_consensus.cc:482] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:54.846700 19209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:54.847429 19209 raft_consensus.cc:504] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a369aefa97e8428d972edd4204277fbd" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 36151 } }
I20260809 09:27:54.847728 19209 leader_election.cc:302] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a369aefa97e8428d972edd4204277fbd; no voters:
I20260809 09:27:54.847985 19209 leader_election.cc:288] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:54.848083 19215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:54.848325 19215 raft_consensus.cc:686] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [term 1 LEADER]: Becoming Leader. State: Replica: a369aefa97e8428d972edd4204277fbd, State: Running, Role: LEADER
I20260809 09:27:54.848625 19215 consensus_queue.cc:237] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a369aefa97e8428d972edd4204277fbd" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 36151 } }
I20260809 09:27:54.849017 19209 sys_catalog.cc:564] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:54.850069 19218 sys_catalog.cc:455] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a369aefa97e8428d972edd4204277fbd. Latest consensus state: current_term: 1 leader_uuid: "a369aefa97e8428d972edd4204277fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a369aefa97e8428d972edd4204277fbd" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 36151 } } }
I20260809 09:27:54.850193 19218 sys_catalog.cc:458] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:54.850546 19233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:54.850521 19216 sys_catalog.cc:455] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a369aefa97e8428d972edd4204277fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a369aefa97e8428d972edd4204277fbd" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 36151 } } }
I20260809 09:27:54.851126 19216 sys_catalog.cc:458] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:54.853458 19233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:54.858131 19233 catalog_manager.cc:1347] Generated new cluster ID: fe0a06a3a2ce4b9f92a328c9f90038d1
I20260809 09:27:54.858189 19233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:54.882864 19233 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:54.883687 19233 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:54.890244 19233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a369aefa97e8428d972edd4204277fbd: Generated new TSK 0
I20260809 09:27:54.890753 19233 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:27:54.921068 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:36151
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:55.021345 19254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:55.021623 19254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:55.021710 19254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:55.025334 19254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:55.025403 19254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:55.025494 19254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:27:55.029922 19254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:55.030781 19254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:55.031656 19254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:55.040252 19263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:55.040277 19262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:55.040252 19266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:55.040535 19254 server_base.cc:1034] running on GCE node
I20260809 09:27:55.040877 19254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:55.041414 19254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:55.042622 19254 hybrid_clock.cc:648] HybridClock initialized: now 1786267675042596 us; error 46 us; skew 500 ppm
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:55.045149 19254 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:27:55.045969 19254 webserver.cc:460] Webserver started at http://127.9.102.1:42825/ using document root <none> and password file <none>
I20260809 09:27:55.046434 19254 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:55.046490 19254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:55.046643 19254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:55.048388 19254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "1768602383604b15b1b28eb480519d58"
format_stamp: "Formatted at 2026-08-09 09:27:55 on dist-test-slave-7vjv"
server_key: "903bea22bed61927cd418108dfa451ce"
server_key_iv: "e21eca4cfbd74bd19d618e74bc744617"
server_key_version: "encryptionkey@0"
I20260809 09:27:55.048882 19254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "1768602383604b15b1b28eb480519d58"
format_stamp: "Formatted at 2026-08-09 09:27:55 on dist-test-slave-7vjv"
server_key: "903bea22bed61927cd418108dfa451ce"
server_key_iv: "e21eca4cfbd74bd19d618e74bc744617"
server_key_version: "encryptionkey@0"
I20260809 09:27:55.052304 19254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260809 09:27:55.054337 19278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:55.055413 19254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:55.055502 19254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "1768602383604b15b1b28eb480519d58"
format_stamp: "Formatted at 2026-08-09 09:27:55 on dist-test-slave-7vjv"
server_key: "903bea22bed61927cd418108dfa451ce"
server_key_iv: "e21eca4cfbd74bd19d618e74bc744617"
server_key_version: "encryptionkey@0"
I20260809 09:27:55.055573 19254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:55.065038 19254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:55.068338 19254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:55.068521 19254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:55.069084 19254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:55.069813 19254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:55.069859 19254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:55.069893 19254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:55.069914 19254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:55.077997 19254 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:44235
I20260809 09:27:55.078018 19452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:44235 every 8 connection(s)
I20260809 09:27:55.078711 19254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:27:55.086575 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 19254
I20260809 09:27:55.086716 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:27:55.087013 9624 external_mini_cluster.cc:1428] Setting key ba11c00894fc330de76bab22f58e7be4
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:55.091946 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:55.080290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:57073 (local address 127.9.102.62:36151)
0809 09:27:55.080444 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:55.080448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:55.081185 (+ 737us) server_negotiation.cc:407] Connection header received
0809 09:27:55.082793 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:55.082796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:55.082839 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:55.082909 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:55.083856 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.084443 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.085758 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.085994 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.086074 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:55.088499 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:55.088502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:55.088505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:55.088537 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:55.090104 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:55.090575 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:55.090579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:55.090581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:55.090665 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:55.090987 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:55.090992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:55.090993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:55.091163 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:55.091280 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:55.091688 (+ 408us) server_negotiation.cc:299] Negotiation successful
0809 09:27:55.091795 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 09:27:55.092797 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:55.080660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:36151 (local address 127.9.102.1:57073)
0809 09:27:55.081053 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:55.081088 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:55.082620 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:55.083046 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:55.083051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:55.083339 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:55.083729 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:55.083740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.084574 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.084578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:55.085643 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:55.085653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.086117 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.086119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:55.086251 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:55.086773 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:55.086789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:55.088368 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:55.090217 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:55.090222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:55.090231 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:55.090463 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:55.090758 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:55.090762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:55.090764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:55.090893 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:55.091279 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:55.091285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:55.091487 (+ 202us) client_negotiation.cc:769] Sending connection context
0809 09:27:55.091648 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 09:27:55.091903 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":105,"threads_started":1}
I20260809 09:27:55.093869 19453 heartbeater.cc:347] Connected to a master server at 127.9.102.62:36151
I20260809 09:27:55.094058 19453 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:55.094543 19453 heartbeater.cc:511] Master 127.9.102.62:36151 requested a full tablet report, sending...
I20260809 09:27:55.095994 19152 ts_manager.cc:194] Registered new tserver with Master: 1768602383604b15b1b28eb480519d58 (127.9.102.1:44235)
I20260809 09:27:55.096993 19152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:57073
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:27:55.112807 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:36151
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:55.211568 19462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:55.211766 19462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:55.211803 19462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:55.215261 19462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:55.215302 19462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:55.215351 19462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:27:55.219682 19462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:55.220438 19462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:55.221343 19462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:55.229478 19473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:55.229478 19471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:55.229493 19469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:55.229799 19462 server_base.cc:1034] running on GCE node
I20260809 09:27:55.230134 19462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:55.230661 19462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:55.231817 19462 hybrid_clock.cc:648] HybridClock initialized: now 1786267675231804 us; error 21 us; skew 500 ppm
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:55.234324 19462 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:27:55.235080 19462 webserver.cc:460] Webserver started at http://127.9.102.2:40429/ using document root <none> and password file <none>
I20260809 09:27:55.235594 19462 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:55.235654 19462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:55.235893 19462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:55.237895 19462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "98af98122268444b9e08ce1444dd4ba4"
format_stamp: "Formatted at 2026-08-09 09:27:55 on dist-test-slave-7vjv"
server_key: "c31c34be7b3f6a484c6a59749a41848f"
server_key_iv: "0efdce88dfdd765a0ba163f1ae9a7264"
server_key_version: "encryptionkey@0"
I20260809 09:27:55.238474 19462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "98af98122268444b9e08ce1444dd4ba4"
format_stamp: "Formatted at 2026-08-09 09:27:55 on dist-test-slave-7vjv"
server_key: "c31c34be7b3f6a484c6a59749a41848f"
server_key_iv: "0efdce88dfdd765a0ba163f1ae9a7264"
server_key_version: "encryptionkey@0"
I20260809 09:27:55.241950 19462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:55.244084 19484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:55.245193 19462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:55.245419 19462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "98af98122268444b9e08ce1444dd4ba4"
format_stamp: "Formatted at 2026-08-09 09:27:55 on dist-test-slave-7vjv"
server_key: "c31c34be7b3f6a484c6a59749a41848f"
server_key_iv: "0efdce88dfdd765a0ba163f1ae9a7264"
server_key_version: "encryptionkey@0"
I20260809 09:27:55.245549 19462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:55.279618 19462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:55.282222 19462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:55.282414 19462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:55.282930 19462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:55.283677 19462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:55.283761 19462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:55.283823 19462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:55.283862 19462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:55.291885 19462 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:38831
I20260809 09:27:55.291913 19680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:38831 every 8 connection(s)
I20260809 09:27:55.292732 19462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:27:55.298650 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 19462
I20260809 09:27:55.298745 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:27:55.298991 9624 external_mini_cluster.cc:1428] Setting key e9361e94511540626640735eb06baea5
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:55.305678 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:55.294366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:53525 (local address 127.9.102.62:36151)
0809 09:27:55.294512 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:55.294516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:55.295228 (+ 712us) server_negotiation.cc:407] Connection header received
0809 09:27:55.296505 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:55.296508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:55.296554 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:55.296635 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:55.297610 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.298121 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.299193 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.299445 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.299564 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:55.301997 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:55.302003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:55.302009 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:55.302038 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:55.303429 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:55.304006 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:55.304012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:55.304016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:55.304139 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:55.304413 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:55.304419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:55.304423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:55.304670 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:55.304769 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:55.305241 (+ 472us) server_negotiation.cc:299] Negotiation successful
0809 09:27:55.305404 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 09:27:55.306361 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:55.294664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:36151 (local address 127.9.102.2:53525)
0809 09:27:55.295087 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:55.295124 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:55.296327 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:55.296779 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:55.296784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:55.297094 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:55.297488 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:55.297500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.298214 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.298216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:55.299075 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:55.299085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.299595 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.299599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:55.299783 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:55.300372 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:55.300389 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:55.301847 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:55.303549 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:55.303557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:55.303571 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:55.303908 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:55.304218 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:55.304220 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:55.304222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:55.304341 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:55.304773 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:55.304778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:55.305004 (+ 226us) client_negotiation.cc:769] Sending connection context
0809 09:27:55.305147 (+ 143us) client_negotiation.cc:240] Negotiation successful
0809 09:27:55.305454 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":107,"threads_started":1}
I20260809 09:27:55.307376 19682 heartbeater.cc:347] Connected to a master server at 127.9.102.62:36151
I20260809 09:27:55.307610 19682 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:55.308154 19682 heartbeater.cc:511] Master 127.9.102.62:36151 requested a full tablet report, sending...
I20260809 09:27:55.309262 19152 ts_manager.cc:194] Registered new tserver with Master: 98af98122268444b9e08ce1444dd4ba4 (127.9.102.2:38831)
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
I20260809 09:27:55.309960 19152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:53525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:27:55.327273 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:36151
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:55.430080 19690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:55.430287 19690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:55.430325 19690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:55.434028 19690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:55.434095 19690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:55.434150 19690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:27:55.439260 19690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:36151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:55.440194 19690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:55.441104 19690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:55.449723 19698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:55.449774 19696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:55.449725 19700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:55.450107 19690 server_base.cc:1034] running on GCE node
I20260809 09:27:55.450518 19690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:55.451143 19690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:55.452337 19690 hybrid_clock.cc:648] HybridClock initialized: now 1786267675452320 us; error 31 us; skew 500 ppm
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:55.454895 19690 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:27:55.455651 19690 webserver.cc:460] Webserver started at http://127.9.102.3:33665/ using document root <none> and password file <none>
I20260809 09:27:55.456169 19690 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:55.456233 19690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:55.456387 19690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:55.458256 19690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "62add4790969440ba031c547aa85bf1e"
format_stamp: "Formatted at 2026-08-09 09:27:55 on dist-test-slave-7vjv"
server_key: "65c833d10fe40f2a2c10752030be7dbd"
server_key_iv: "fa59e6212476f5eaf3608d0f6e1d74ef"
server_key_version: "encryptionkey@0"
I20260809 09:27:55.458765 19690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "62add4790969440ba031c547aa85bf1e"
format_stamp: "Formatted at 2026-08-09 09:27:55 on dist-test-slave-7vjv"
server_key: "65c833d10fe40f2a2c10752030be7dbd"
server_key_iv: "fa59e6212476f5eaf3608d0f6e1d74ef"
server_key_version: "encryptionkey@0"
I20260809 09:27:55.462339 19690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:55.464365 19711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:55.465409 19690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:55.465498 19690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "62add4790969440ba031c547aa85bf1e"
format_stamp: "Formatted at 2026-08-09 09:27:55 on dist-test-slave-7vjv"
server_key: "65c833d10fe40f2a2c10752030be7dbd"
server_key_iv: "fa59e6212476f5eaf3608d0f6e1d74ef"
server_key_version: "encryptionkey@0"
I20260809 09:27:55.465586 19690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:55.476770 19690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:55.478948 19690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:55.479077 19690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:55.479482 19690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:55.480196 19690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:55.480238 19690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:55.480270 19690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:55.480288 19690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:55.488281 19690 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:45157
I20260809 09:27:55.488318 19904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:45157 every 8 connection(s)
I20260809 09:27:55.489006 19690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:27:55.492815 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 19690
I20260809 09:27:55.492923 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:27:55.493192 9624 external_mini_cluster.cc:1428] Setting key 4fe219fb25ce2500063a5f0a1a945797
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:27:55.501523 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:55.490624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:58145 (local address 127.9.102.62:36151)
0809 09:27:55.490755 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:55.490758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:55.491467 (+ 709us) server_negotiation.cc:407] Connection header received
0809 09:27:55.492833 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:55.492836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:55.492891 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:55.492974 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:55.494331 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.494938 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.496068 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.496312 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.496388 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:55.498423 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:55.498426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:55.498427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:55.498448 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:55.499811 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:55.500312 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:55.500314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:55.500315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:55.500422 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:55.500710 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:55.500713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:55.500714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:55.500869 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:55.500950 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:55.501333 (+ 383us) server_negotiation.cc:299] Negotiation successful
0809 09:27:55.501418 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 09:27:55.502250 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:55.490928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:36151 (local address 127.9.102.3:58145)
0809 09:27:55.491336 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:55.491373 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:55.492639 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:55.493201 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:55.493210 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:55.493609 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:55.494179 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:55.494198 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.495074 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.495078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:55.495957 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:55.495967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.496438 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.496441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:55.496572 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:55.496991 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:55.497005 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:55.498289 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:55.499903 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:55.499907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:55.499916 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:55.500220 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:55.500500 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:55.500502 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:55.500503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:55.500623 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:55.500960 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:55.500964 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:55.501156 (+ 192us) client_negotiation.cc:769] Sending connection context
0809 09:27:55.501289 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 09:27:55.501485 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":110,"threads_started":1}
I20260809 09:27:55.503244 19906 heartbeater.cc:347] Connected to a master server at 127.9.102.62:36151
I20260809 09:27:55.503434 19906 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:55.503837 19906 heartbeater.cc:511] Master 127.9.102.62:36151 requested a full tablet report, sending...
I20260809 09:27:55.504874 19152 ts_manager.cc:194] Registered new tserver with Master: 62add4790969440ba031c547aa85bf1e (127.9.102.3:45157)
I20260809 09:27:55.505406 19152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:58145
I20260809 09:27:55.506953 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/test-admin.keytab.
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:55.522141 9624 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 09:27:55.524617 19919 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:27:55.523277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.9.102.1:44235)
0809 09:27:55.523617 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:55.523626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:55.523646 (+ 20us) server_negotiation.cc:407] Connection header received
0809 09:27:55.523687 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:55.523691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:55.523827 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:55.523930 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:55.524441 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53486: BlockingRecv error: recv got EOF from 127.0.0.1:53486 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":131,"threads_started":1}
W20260809 09:27:55.524632 9624 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/krb5cc' of type FILE
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:55.525548 9624 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:55.526365 9624 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.1@KRBTEST.COM
I20260809 09:27:55.534700 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:55.527052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.9.102.1:44235)
0809 09:27:55.527157 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:55.527162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:55.527231 (+ 69us) server_negotiation.cc:407] Connection header received
0809 09:27:55.527353 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:55.527356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:55.527397 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:55.527468 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:55.527915 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.528646 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.529354 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.529667 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.529774 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:55.531621 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:55.531629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:55.531639 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:55.531661 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:55.533020 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:55.533455 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:55.533460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:55.533462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:55.533544 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:55.533810 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:55.533814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:55.533816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:55.534032 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:55.534217 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:55.534443 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 09:27:55.534595 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 09:27:55.537843 19919 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:27:55.536960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.9.102.1:44235)
0809 09:27:55.537076 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:55.537079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:55.537163 (+ 84us) server_negotiation.cc:407] Connection header received
0809 09:27:55.537301 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:55.537303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:55.537337 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:55.537403 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:55.537747 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53512: BlockingRecv error: recv got EOF from 127.0.0.1:53512 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260809 09:27:55.537964 9624 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/krb5cc' of type FILE
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:55.538789 9624 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:55.539549 9624 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.1@KRBTEST.COM
I20260809 09:27:55.546530 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:55.540213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.9.102.1:44235)
0809 09:27:55.540366 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:55.540369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:55.540425 (+ 56us) server_negotiation.cc:407] Connection header received
0809 09:27:55.540521 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:55.540529 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:55.540567 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:55.540641 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:55.541036 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.541450 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.542121 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.542304 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.542371 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:55.544309 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:55.544311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:55.544313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:55.544331 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:55.545361 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:55.545675 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:55.545677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:55.545678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:55.545738 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:55.545930 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:55.545931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:55.545932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:55.546049 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:55.546127 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:55.546288 (+ 161us) server_negotiation.cc:299] Negotiation successful
0809 09:27:55.546389 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260809 09:27:55.548668 19919 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:27:55.547861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.9.102.1:44235)
0809 09:27:55.547969 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:55.547974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:55.547989 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:27:55.548077 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:55.548079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:55.548161 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:55.548225 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:55.548598 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53520: BlockingRecv error: recv got EOF from 127.0.0.1:53520 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 09:27:55.548766 9624 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786267646682796-9624-0/krb5kdc/krb5cc' of type FILE
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:55.549619 9624 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:55.550484 9624 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.1@KRBTEST.COM
I20260809 09:27:55.557101 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:55.551106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.9.102.1:44235)
0809 09:27:55.551217 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:55.551220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:55.551231 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:27:55.551300 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:55.551302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:55.551327 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:55.551375 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:55.551728 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.552134 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.552779 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:55.552940 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:55.553002 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:55.554753 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:55.554755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:55.554757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:55.554774 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:55.555732 (+ 958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:55.556053 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:55.556055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:55.556072 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:55.556205 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:55.556434 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:55.556436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:55.556437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:55.556559 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:55.556645 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:55.556845 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 09:27:55.556971 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 09:27:55.557958 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 19254
I20260809 09:27:55.575585 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 19462
I20260809 09:27:55.591849 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 19690
I20260809 09:27:55.599454 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 19100
2026-08-09T09:27:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2202 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19941](info): set up 2 sockets
Aug 09 09:27:55 dist-test-slave-7vjv krb5kdc[19941](info): commencing operation
krb5kdc: starting...
W20260809 09:27:56.820585 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:27:56 dist-test-slave-7vjv krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:27:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:27:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.62.keytab.
Entry for principal oryx/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.62.keytab.
I20260809 09:27:56.926445 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:37729
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.62.keytab
--principal=oryx/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:37729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:57.026907 19957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:57.027114 19957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:57.027149 19957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:57.030694 19957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:57.030743 19957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:57.030763 19957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:57.030779 19957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:57.030794 19957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:57.035379 19957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:37729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.62.keytab
--principal=oryx/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:37729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:57.036262 19957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:57.037115 19957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:57.044543 19965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:57.044559 19971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:57.044596 19973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:57.044842 19957 server_base.cc:1034] running on GCE node
I20260809 09:27:57.045408 19957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:57.046353 19957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:57.047569 19957 hybrid_clock.cc:648] HybridClock initialized: now 1786267677047544 us; error 34 us; skew 500 ppm
Aug 09 09:27:57 dist-test-slave-7vjv krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267677, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:57.050244 19957 init.cc:377] Logged in from keytab as oryx/127.9.102.62@KRBTEST.COM (short username oryx)
I20260809 09:27:57.051071 19957 webserver.cc:460] Webserver started at http://127.9.102.62:46439/ using document root <none> and password file <none>
I20260809 09:27:57.051558 19957 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:57.051642 19957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:57.051849 19957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:57.053630 19957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "c1ef11a47c3845b09d02d5a1472c860e"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "70ed853983c31e7c2a46a51e58859959"
server_key_iv: "2af011efa00223c91a5b314ef2abd8d8"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.054136 19957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "c1ef11a47c3845b09d02d5a1472c860e"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "70ed853983c31e7c2a46a51e58859959"
server_key_iv: "2af011efa00223c91a5b314ef2abd8d8"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.057271 19957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:27:57.059269 19986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:57.060413 19957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 09:27:57.060544 19957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "c1ef11a47c3845b09d02d5a1472c860e"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "70ed853983c31e7c2a46a51e58859959"
server_key_iv: "2af011efa00223c91a5b314ef2abd8d8"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.060644 19957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:57.079280 19957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:57.081693 19957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:57.081878 19957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:57.088074 19957 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:37729
I20260809 09:27:57.088361 20072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:37729 every 8 connection(s)
I20260809 09:27:57.088894 19957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:27:57.091253 20073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:57.091353 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 19957
I20260809 09:27:57.091457 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:27:57.091766 9624 external_mini_cluster.cc:1428] Setting key 5ac7af13a9e93456006c8f3472afb373
I20260809 09:27:57.097216 20073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e: Bootstrap starting.
I20260809 09:27:57.099666 20073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e: Neither blocks nor log segments found. Creating new log.
Aug 09 09:27:57 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:27:57.100510 20073 log.cc:824] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:57.102335 20073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e: No bootstrap required, opened a new log
I20260809 09:27:57.104743 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.093007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.9.102.62:37729)
0809 09:27:57.093381 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:57.093396 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:57.093427 (+ 31us) server_negotiation.cc:407] Connection header received
0809 09:27:57.094075 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:57.094090 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:57.094370 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:57.094606 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:57.095403 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.096282 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.097382 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.097801 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.097937 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.100778 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:57.100783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:57.100794 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:57.100819 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:57.102360 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.102789 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.102793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.102797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.102905 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.103158 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.103162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.103163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.103480 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:57.103655 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:57.103915 (+ 260us) server_negotiation.cc:299] Negotiation successful
0809 09:27:57.104193 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":121,"threads_started":1}
I20260809 09:27:57.105345 20073 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } }
I20260809 09:27:57.105554 20073 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:57.105614 20073 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1ef11a47c3845b09d02d5a1472c860e, State: Initialized, Role: FOLLOWER
I20260809 09:27:57.105962 20073 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } }
I20260809 09:27:57.106096 20073 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:27:57.106171 20073 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:27:57.106283 20073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:57.107085 20073 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } }
I20260809 09:27:57.107381 20073 leader_election.cc:302] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1ef11a47c3845b09d02d5a1472c860e; no voters:
I20260809 09:27:57.107640 20073 leader_election.cc:288] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:27:57.107748 20079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:57.107965 20079 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 1 LEADER]: Becoming Leader. State: Replica: c1ef11a47c3845b09d02d5a1472c860e, State: Running, Role: LEADER
I20260809 09:27:57.108230 20079 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } }
I20260809 09:27:57.108719 20073 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:57.109702 20084 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1ef11a47c3845b09d02d5a1472c860e. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } }
I20260809 09:27:57.109820 20084 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:57.110031 20082 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } }
I20260809 09:27:57.110113 20082 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:57.110447 20098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:27:57.113103 20098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:27:57.117700 20098 catalog_manager.cc:1347] Generated new cluster ID: 893b9bc4eb0343f08d68c4feac02e2a7
I20260809 09:27:57.117754 20098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:27:57.126157 20098 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:27:57.127135 20098 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:27:57.134137 20098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e: Generated new TSK 0
I20260809 09:27:57.134696 20098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab.
Entry for principal oryx/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab.
I20260809 09:27:57.160907 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:37729
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab
--principal=oryx/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:57.260411 20117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:57.260620 20117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:57.260656 20117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:57.264194 20117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:57.264240 20117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:57.264292 20117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:27:57.268855 20117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab
--principal=oryx/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:37729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:57.269622 20117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:57.270390 20117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:57.278648 20127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:57.278648 20126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:57.278667 20131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:57.278950 20117 server_base.cc:1034] running on GCE node
I20260809 09:27:57.279276 20117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:57.279795 20117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:57.280936 20117 hybrid_clock.cc:648] HybridClock initialized: now 1786267677280923 us; error 22 us; skew 500 ppm
Aug 09 09:27:57 dist-test-slave-7vjv krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267677, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:57.283387 20117 init.cc:377] Logged in from keytab as oryx/127.9.102.1@KRBTEST.COM (short username oryx)
I20260809 09:27:57.284207 20117 webserver.cc:460] Webserver started at http://127.9.102.1:43003/ using document root <none> and password file <none>
I20260809 09:27:57.284690 20117 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:57.284742 20117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:57.284883 20117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:57.286568 20117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "060f8bc83055420a98a44897a75f6fb1"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "97a6f49ed0fab4f8ff931cdcd0de30e1"
server_key_iv: "31841f7d57aed73ad71abb6e949c9327"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.287042 20117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "060f8bc83055420a98a44897a75f6fb1"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "97a6f49ed0fab4f8ff931cdcd0de30e1"
server_key_iv: "31841f7d57aed73ad71abb6e949c9327"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.290303 20117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:27:57.292464 20139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:57.293485 20117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:57.293612 20117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "060f8bc83055420a98a44897a75f6fb1"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "97a6f49ed0fab4f8ff931cdcd0de30e1"
server_key_iv: "31841f7d57aed73ad71abb6e949c9327"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.293710 20117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:57.317709 20117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:57.320159 20117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:57.320331 20117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:57.320822 20117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:57.321513 20117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:57.321587 20117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:57.321658 20117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:57.321705 20117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:57.329753 20117 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:38807
I20260809 09:27:57.329780 20325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:38807 every 8 connection(s)
I20260809 09:27:57.330451 20117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:27:57.336243 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 20117
I20260809 09:27:57.336362 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:27:57.336659 9624 external_mini_cluster.cc:1428] Setting key bd8cdeb4fad09ed2d5b936f6faf41acb
Aug 09 09:27:57 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267677, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:27:57.344852 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.332048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:38207 (local address 127.9.102.62:37729)
0809 09:27:57.332206 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:57.332210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:57.333171 (+ 961us) server_negotiation.cc:407] Connection header received
0809 09:27:57.334092 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:57.334111 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:57.334270 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:57.334343 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:57.335514 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.335990 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.337681 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.338016 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.338152 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.341015 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:57.341018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:57.341020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:57.341044 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:57.342680 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.343165 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.343172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.343176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.343282 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.343627 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.343632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.343636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.343902 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:57.344007 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:57.344490 (+ 483us) server_negotiation.cc:299] Negotiation successful
0809 09:27:57.344636 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 09:27:57.345283 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.332381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.1:38207)
0809 09:27:57.333004 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:57.333053 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:57.333872 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:57.334539 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:57.334547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:57.334940 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:57.335393 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.335405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.336128 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.336133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.337540 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.337551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.338189 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.338193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.338383 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.338934 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:57.338957 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:57.340840 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:57.342796 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.342801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.342811 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.343049 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.343383 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.343385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.343386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.343506 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.344009 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:57.344013 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:57.344250 (+ 237us) client_negotiation.cc:769] Sending connection context
0809 09:27:57.344379 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 09:27:57.344594 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":108,"threads_started":1}
I20260809 09:27:57.346340 20327 heartbeater.cc:347] Connected to a master server at 127.9.102.62:37729
I20260809 09:27:57.346567 20327 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:57.347059 20327 heartbeater.cc:511] Master 127.9.102.62:37729 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:27:57.348537 20011 ts_manager.cc:194] Registered new tserver with Master: 060f8bc83055420a98a44897a75f6fb1 (127.9.102.1:38807)
I20260809 09:27:57.349674 20011 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:38207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab.
Entry for principal oryx/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab.
I20260809 09:27:57.364403 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:37729
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab
--principal=oryx/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:57.463323 20334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:57.463599 20334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:57.463670 20334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:57.467331 20334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:57.467408 20334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:57.467494 20334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:27:57.472317 20334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab
--principal=oryx/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:37729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:57.473228 20334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:57.474172 20334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:57.483180 20345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:57.483197 20349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:57.483333 20334 server_base.cc:1034] running on GCE node
W20260809 09:27:57.483197 20352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:57.483880 20334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:57.484481 20334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:57.485659 20334 hybrid_clock.cc:648] HybridClock initialized: now 1786267677485642 us; error 29 us; skew 500 ppm
Aug 09 09:27:57 dist-test-slave-7vjv krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267677, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:57.488217 20334 init.cc:377] Logged in from keytab as oryx/127.9.102.2@KRBTEST.COM (short username oryx)
I20260809 09:27:57.489028 20334 webserver.cc:460] Webserver started at http://127.9.102.2:38339/ using document root <none> and password file <none>
I20260809 09:27:57.489526 20334 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:57.489610 20334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:57.489820 20334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:57.491508 20334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "84f801fb831045dab3d877042086156a"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "97eabb0684f33aff9243fabec0832321"
server_key_iv: "4090936cec3527bb0b4e10114cdcc1f9"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.492027 20334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "84f801fb831045dab3d877042086156a"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "97eabb0684f33aff9243fabec0832321"
server_key_iv: "4090936cec3527bb0b4e10114cdcc1f9"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.495249 20334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 09:27:57.497345 20362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:57.498430 20334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:27:57.498538 20334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "84f801fb831045dab3d877042086156a"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "97eabb0684f33aff9243fabec0832321"
server_key_iv: "4090936cec3527bb0b4e10114cdcc1f9"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.498637 20334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:57.521976 20334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:57.524508 20334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:57.524674 20334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:57.525152 20334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:57.525830 20334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:57.525894 20334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:57.525954 20334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:57.526000 20334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:57.533985 20334 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:34507
I20260809 09:27:57.534029 20537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:34507 every 8 connection(s)
I20260809 09:27:57.534680 20334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:27:57.540198 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 20334
I20260809 09:27:57.540326 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:27:57.540607 9624 external_mini_cluster.cc:1428] Setting key bdc0912caed910d5b869d094eaa9090b
Aug 09 09:27:57 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267677, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:27:57.547181 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.536182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:53767 (local address 127.9.102.62:37729)
0809 09:27:57.536311 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:57.536314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:57.537142 (+ 828us) server_negotiation.cc:407] Connection header received
0809 09:27:57.537890 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:57.537892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:57.537932 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:57.537995 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:57.538997 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.539507 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.540889 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.541171 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.541275 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.543644 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:57.543652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:57.543654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:57.543675 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:57.545226 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.545672 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.545678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.545682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.545772 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.546062 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.546067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.546071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.546296 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:57.546391 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:57.546773 (+ 382us) server_negotiation.cc:299] Negotiation successful
0809 09:27:57.546914 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 09:27:57.547694 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.536469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.2:53767)
0809 09:27:57.537037 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:57.537070 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:57.537725 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:57.538153 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:57.538159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:57.538454 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:57.538857 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.538870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.539631 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.539634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.540759 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.540768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.541354 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.541358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.541546 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.542144 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:57.542164 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:57.543491 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:57.545347 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.545351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.545360 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.545569 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.545858 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.545860 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.545866 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.545972 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.546388 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:57.546392 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:57.546560 (+ 168us) client_negotiation.cc:769] Sending connection context
0809 09:27:57.546685 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 09:27:57.546891 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":93,"threads_started":1}
I20260809 09:27:57.548803 20539 heartbeater.cc:347] Connected to a master server at 127.9.102.62:37729
I20260809 09:27:57.549038 20539 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:57.549525 20539 heartbeater.cc:511] Master 127.9.102.62:37729 requested a full tablet report, sending...
I20260809 09:27:57.550537 20011 ts_manager.cc:194] Registered new tserver with Master: 84f801fb831045dab3d877042086156a (127.9.102.2:34507)
I20260809 09:27:57.551095 20011 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:53767
WARNING: no policy specified for oryx/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab.
Entry for principal oryx/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab.
I20260809 09:27:57.568429 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:37729
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab
--principal=oryx/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:57.670106 20547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:57.670305 20547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:57.670341 20547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:57.674026 20547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:57.674077 20547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:57.674129 20547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:27:57.678390 20547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab
--principal=oryx/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:37729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:57.679207 20547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:57.680135 20547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:57.688691 20554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:57.688663 20552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:57.688663 20556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:57.688824 20547 server_base.cc:1034] running on GCE node
I20260809 09:27:57.689181 20547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:57.689679 20547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:57.690860 20547 hybrid_clock.cc:648] HybridClock initialized: now 1786267677690839 us; error 40 us; skew 500 ppm
Aug 09 09:27:57 dist-test-slave-7vjv krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267677, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:57.693400 20547 init.cc:377] Logged in from keytab as oryx/127.9.102.3@KRBTEST.COM (short username oryx)
I20260809 09:27:57.694075 20547 webserver.cc:460] Webserver started at http://127.9.102.3:33367/ using document root <none> and password file <none>
I20260809 09:27:57.694521 20547 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:57.694612 20547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:57.694808 20547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:57.696563 20547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "1228037bffae4bb1be226e7a9c103953"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "74963e99f46f062819fda6a3805f15cd"
server_key_iv: "0da7d212a286f4e3f1bb1efbc3439ce5"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.697057 20547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "1228037bffae4bb1be226e7a9c103953"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "74963e99f46f062819fda6a3805f15cd"
server_key_iv: "0da7d212a286f4e3f1bb1efbc3439ce5"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.700322 20547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:27:57.702337 20571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:57.703449 20547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:57.703585 20547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "1228037bffae4bb1be226e7a9c103953"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "74963e99f46f062819fda6a3805f15cd"
server_key_iv: "0da7d212a286f4e3f1bb1efbc3439ce5"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.703678 20547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:57.720683 20547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:57.723516 20547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:57.723680 20547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:57.724172 20547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:57.724880 20547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:57.724943 20547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:57.725001 20547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:57.725045 20547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:57.732846 20547 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:41209
I20260809 09:27:57.732874 20745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:41209 every 8 connection(s)
I20260809 09:27:57.733809 20547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
Aug 09 09:27:57 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267677, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:27:57.743912 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 20547
I20260809 09:27:57.744024 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:27:57.744336 9624 external_mini_cluster.cc:1428] Setting key 5ebc14b3de452c0233d78c89aa753fe7
I20260809 09:27:57.745816 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.735476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:41407 (local address 127.9.102.62:37729)
0809 09:27:57.735613 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:57.735617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:57.736470 (+ 853us) server_negotiation.cc:407] Connection header received
0809 09:27:57.737195 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:57.737198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:57.737245 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:57.737319 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:57.738294 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.738778 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.739801 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.740009 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.740160 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.742284 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:57.742287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:57.742289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:57.742312 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:57.743707 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.744224 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.744229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.744232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.744322 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.744693 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.744695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.744697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.744908 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:57.744994 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:57.745586 (+ 592us) server_negotiation.cc:299] Negotiation successful
0809 09:27:57.745701 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 09:27:57.746507 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.735745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.3:41407)
0809 09:27:57.736362 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:57.736395 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:57.737033 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:57.737460 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:57.737465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:57.737786 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:57.738164 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.738175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.738885 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.738887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.739668 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.739677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.740190 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.740193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.740328 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.740788 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:57.740802 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:57.742139 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:57.743812 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.743816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.743824 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.744123 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.744430 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.744434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.744436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.744570 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.745032 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:57.745038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:57.745320 (+ 282us) client_negotiation.cc:769] Sending connection context
0809 09:27:57.745488 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 09:27:57.745777 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":102,"threads_started":1}
I20260809 09:27:57.747401 20746 heartbeater.cc:347] Connected to a master server at 127.9.102.62:37729
I20260809 09:27:57.747571 20746 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:57.747913 20746 heartbeater.cc:511] Master 127.9.102.62:37729 requested a full tablet report, sending...
I20260809 09:27:57.748772 20011 ts_manager.cc:194] Registered new tserver with Master: 1228037bffae4bb1be226e7a9c103953 (127.9.102.3:41209)
I20260809 09:27:57.749238 20011 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:41407
I20260809 09:27:57.757843 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.102.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.61.keytab.
Entry for principal oryx/127.9.102.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.61.keytab.
I20260809 09:27:57.782743 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.61:46537
--webserver_interface=127.9.102.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.61.keytab
--principal=oryx/127.9.102.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:37729,127.9.102.61:46537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:27:57.881016 20752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:57.881215 20752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:57.881273 20752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:57.884934 20752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:57.884979 20752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:57.884995 20752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:57.885010 20752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:57.885059 20752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:57.889916 20752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:37729,127.9.102.61:46537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.61.keytab
--principal=oryx/127.9.102.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.61:46537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.102.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
W20260809 09:27:57.890100 20752 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.102.62:37729,127.9.102.61:46537'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 09:27:57.890770 20752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:57.891579 20752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:57.899214 20757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:57.899219 20759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:57.899214 20763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:57.899325 20752 server_base.cc:1034] running on GCE node
I20260809 09:27:57.899972 20752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:57.900990 20752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:57.902145 20752 hybrid_clock.cc:648] HybridClock initialized: now 1786267677902111 us; error 50 us; skew 500 ppm
Aug 09 09:27:57 dist-test-slave-7vjv krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267677, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:57.904585 20752 init.cc:377] Logged in from keytab as oryx/127.9.102.61@KRBTEST.COM (short username oryx)
I20260809 09:27:57.905347 20752 webserver.cc:460] Webserver started at http://127.9.102.61:39969/ using document root <none> and password file <none>
I20260809 09:27:57.905810 20752 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:57.905864 20752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:57.906090 20752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:57.907707 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data/instance:
uuid: "edc35643011b4a2e8ee9a33dac0243d2"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "6a6aff1da3ae25f98b5814474d0751b2"
server_key_iv: "b9a8d637f15799f7f9ff59eecac821a8"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.908268 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/wal/instance:
uuid: "edc35643011b4a2e8ee9a33dac0243d2"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "6a6aff1da3ae25f98b5814474d0751b2"
server_key_iv: "b9a8d637f15799f7f9ff59eecac821a8"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.911441 20752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260809 09:27:57.913622 20775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:57.914615 20752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 09:27:57.914741 20752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/wal
uuid: "edc35643011b4a2e8ee9a33dac0243d2"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "6a6aff1da3ae25f98b5814474d0751b2"
server_key_iv: "b9a8d637f15799f7f9ff59eecac821a8"
server_key_version: "encryptionkey@0"
I20260809 09:27:57.914832 20752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:57.934089 20752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:57.936355 20752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:57.936518 20752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:57.942615 20752 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.61:46537
I20260809 09:27:57.942881 20857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.61:46537 every 8 connection(s)
I20260809 09:27:57.943429 20752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
I20260809 09:27:57.945845 20858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:57.947974 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 20752
I20260809 09:27:57.949184 20858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 09:27:57 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267677, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.61@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
Aug 09 09:27:57 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.102.61@KRBTEST.COM
I20260809 09:27:57.960204 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.950710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.9.102.62:37729)
0809 09:27:57.950840 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:57.950844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:57.951062 (+ 218us) server_negotiation.cc:407] Connection header received
0809 09:27:57.951298 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:57.951302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:57.951367 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:57.951488 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:57.952318 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.952823 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.953862 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.954064 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.954138 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.956445 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:57.956450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:57.956455 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:57.956485 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:57.957977 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.958578 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.958581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.958582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.958661 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.958978 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.958980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.958981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.959266 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:57.959332 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:57.959866 (+ 534us) server_negotiation.cc:299] Negotiation successful
0809 09:27:57.959985 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 09:27:57.960950 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.949286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.9.102.61:46537)
0809 09:27:57.949719 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:57.949734 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:57.949763 (+ 29us) server_negotiation.cc:407] Connection header received
0809 09:27:57.950286 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:57.950300 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:57.950649 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:57.951121 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:57.952003 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.952994 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.953792 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.954180 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.954318 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.956893 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:57.956898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:57.956905 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:57.956924 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:57.958475 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.958890 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.958895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.958896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.958966 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.959210 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.959212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.959213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.959656 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:57.959822 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:57.960047 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 09:27:57.960501 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":304,"spinlock_wait_cycles":8960,"thread_start_us":116,"threads_started":1}
I20260809 09:27:57.960950 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.950571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:57904)
0809 09:27:57.950942 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:57.950990 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:57.951142 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:57.951555 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:57.951559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:57.951837 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:57.952184 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.952197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.952961 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.952966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.953723 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.953753 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.954192 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.954196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.954321 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.954966 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:57.954980 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:57.956292 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:57.958081 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.958087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.958094 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.958475 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.958747 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.958750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.958751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.958890 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.959353 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:57.959358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:57.959656 (+ 298us) client_negotiation.cc:769] Sending connection context
0809 09:27:57.959808 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 09:27:57.960111 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":121,"spinlock_wait_cycles":11264,"thread_start_us":103,"threads_started":1}
I20260809 09:27:57.962666 20858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 09:27:57 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267677, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.61@KRBTEST.COM for oryx/127.9.102.61@KRBTEST.COM
I20260809 09:27:57.969134 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.963026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:46537 (local address 127.0.0.1:33338)
0809 09:27:57.963128 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:57.963141 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:57.963206 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:57.963371 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:57.963373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:57.963490 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:57.963581 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.963585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.964254 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.964257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.964658 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.964664 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.965557 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.965560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.965620 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.965938 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:57.965947 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:57.966862 (+ 915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:57.968056 (+ 1194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.968058 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.968059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.968278 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.968501 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.968503 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.968504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.968577 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.968819 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:57.968821 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:57.968882 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 09:27:57.968951 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 09:27:57.969036 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 09:27:57.969200 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.963102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.9.102.61:46537)
0809 09:27:57.963198 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:57.963201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:57.963211 (+ 10us) server_negotiation.cc:407] Connection header received
0809 09:27:57.963257 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:57.963259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:57.963288 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:57.963345 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:57.963666 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.964173 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.964750 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.965460 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.965540 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.966973 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:57.966976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:57.966977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:57.966995 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:57.967950 (+ 955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.968355 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.968357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.968358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.968426 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.968629 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.968631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.968632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.968752 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:57.968813 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:57.968977 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 09:27:57.969096 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":34}
I20260809 09:27:57.971469 20858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: Bootstrap starting.
I20260809 09:27:57.973503 20858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:57.974100 20858 log.cc:824] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:57.975584 20858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: No bootstrap required, opened a new log
I20260809 09:27:57.977982 20858 raft_consensus.cc:348] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } }
I20260809 09:27:57.978145 20858 raft_consensus.cc:374] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:57.978174 20858 raft_consensus.cc:729] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc35643011b4a2e8ee9a33dac0243d2, State: Initialized, Role: FOLLOWER
I20260809 09:27:57.978464 20858 consensus_queue.cc:260] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } }
I20260809 09:27:57.978803 20872 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } } }
I20260809 09:27:57.978916 20872 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:27:57.979354 20858 sys_catalog.cc:564] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 09:27:57.983688 20889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:27:57.983762 20889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:27:57.985450 20752 master_runner.cc:428] Detected that this master edc35643011b4a2e8ee9a33dac0243d2 is joining an existing cluster
I20260809 09:27:57.985515 20752 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.102.61:46537
I20260809 09:27:57.994771 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.987627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:46537 (local address 127.0.0.1:33348)
0809 09:27:57.987927 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:57.987940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:57.988060 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:57.988360 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:57.988362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:57.988563 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:57.988666 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.988673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.989476 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.989480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.989968 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.989975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.990465 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.990469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.990556 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.990948 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:57.990961 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:57.991338 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:57.993322 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.993327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.993330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.993523 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.993842 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.993847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.993852 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.993931 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.994293 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:57.994295 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:57.994367 (+ 72us) client_negotiation.cc:769] Sending connection context
0809 09:27:57.994472 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 09:27:57.994601 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":82,"threads_started":1}
I20260809 09:27:57.994758 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.987455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.9.102.61:46537)
0809 09:27:57.987552 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:57.987556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:57.988002 (+ 446us) server_negotiation.cc:407] Connection header received
0809 09:27:57.988178 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:57.988181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:57.988217 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:57.988259 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:57.988779 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.989382 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.990077 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.990346 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.990430 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.991461 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:57.991464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:57.991466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:57.991485 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:57.993211 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.993659 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.993663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.993665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.993749 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.994021 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.994025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.994027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.994189 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:57.994301 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:57.994514 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 09:27:57.994650 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":35}
I20260809 09:27:57.995215 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.987435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.9.102.62:37729)
0809 09:27:57.987548 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:57.987551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:57.987830 (+ 279us) server_negotiation.cc:407] Connection header received
0809 09:27:57.987984 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:57.987987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:57.988029 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:57.988127 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:57.988521 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.989172 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.990258 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.990555 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.990653 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.991844 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:57.991847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:57.991850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:57.991875 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:57.993503 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.993903 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.993906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.993908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.993987 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:57.994317 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:57.994320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:57.994322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:57.994550 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:57.994625 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:57.994968 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 09:27:57.995085 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 09:27:57.995656 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.987385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:57920)
0809 09:27:57.987752 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:57.987767 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:57.987893 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:57.988162 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:57.988164 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:57.988324 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:57.988409 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.988414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.989288 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.989291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.990147 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:57.990157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:57.990665 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:57.990668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:57.990963 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:57.991377 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:57.991392 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:57.991710 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:57.993616 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.993622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.993626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.993822 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.994096 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:57.994099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:57.994100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:57.994197 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:57.994696 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:57.994699 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:57.994792 (+ 93us) client_negotiation.cc:769] Sending connection context
0809 09:27:57.995368 (+ 576us) client_negotiation.cc:240] Negotiation successful
0809 09:27:57.995497 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":93,"threads_started":1}
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267677, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.62@KRBTEST.COM for oryx/127.9.102.61@KRBTEST.COM
I20260809 09:27:58.008620 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.999821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:46537 (local address 127.0.0.1:33350)
0809 09:27:58.000166 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.000213 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.000334 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.000604 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.000610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.000993 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.001119 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.001126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.001791 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.001794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.002797 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.002809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.003219 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.003223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.003329 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.003778 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.003792 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.005203 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.006926 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.006935 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.006940 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.007277 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.007536 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.007541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.007544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.007635 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.007989 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.008008 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.008167 (+ 159us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.008301 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.008481 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
I20260809 09:27:58.008639 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:57.999904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.9.102.61:46537)
0809 09:27:58.000199 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.000203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.000270 (+ 67us) server_negotiation.cc:407] Connection header received
0809 09:27:58.000449 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.000452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.000489 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.000552 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.001226 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.001694 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.002903 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.003114 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.003189 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.005327 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.005330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.005334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.005357 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.006810 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.007370 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.007375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.007381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.007448 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.007731 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.007733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.007735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.007889 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.007987 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.008347 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.008461 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260809 09:27:58.009951 20011 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.102.61:46537: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2"
member_type: NON_VOTER
last_known_addr {
host: "127.9.102.61"
port: 46537
}
attrs {
promote: true
}
}
dest_uuid: "c1ef11a47c3845b09d02d5a1472c860e"
cas_config_opid_index: -1
I20260809 09:27:58.010586 20011 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true } }
I20260809 09:27:58.012151 20909 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER edc35643011b4a2e8ee9a33dac0243d2 (127.9.102.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true } } }
I20260809 09:27:58.012804 20819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:58.013731 20909 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.102.61:46537
I20260809 09:27:58.013914 20909 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true } } }
I20260809 09:27:58.014000 20909 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:58.014225 20819 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 FOLLOWER]: Refusing update from remote peer c1ef11a47c3845b09d02d5a1472c860e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 09:27:58.014420 20752 master.cc:537] Master@127.9.102.61:46537 shutting down...
I20260809 09:27:58.015549 20906 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 09:27:58.017568 19987 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260809 09:27:58.018594 19987 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e -> Peer edc35643011b4a2e8ee9a33dac0243d2 (127.9.102.61:46537): Couldn't send request to peer edc35643011b4a2e8ee9a33dac0243d2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 09:27:58.019480 20752 raft_consensus.cc:2229] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:58.019779 20752 raft_consensus.cc:2258] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:58.019897 20752 tablet_replica.cc:330] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: stopping tablet replica
I20260809 09:27:58.023931 20752 master.cc:552] Master@127.9.102.61:46537 shutdown complete.
I20260809 09:27:58.023990 20752 master_runner.cc:305] Clearing existing system tablet
I20260809 09:27:58.024261 20752 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:58.024333 20752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:58.025398 20752 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 09:27:58.026203 20917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.026499 20752 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:58.026688 20752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
WARNING: no policy specified for oryx/127.9.102.60@KRBTEST.COM; defaulting to no policy
I20260809 09:27:58.026739 20752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/wal
uuid: "edc35643011b4a2e8ee9a33dac0243d2"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "6a6aff1da3ae25f98b5814474d0751b2"
server_key_iv: "b9a8d637f15799f7f9ff59eecac821a8"
server_key_version: "encryptionkey@0"
I20260809 09:27:58.028443 20752 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.60@KRBTEST.COM" created.
I20260809 09:27:58.031188 20752 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 09:27:58.031306 20752 log.cc:1196] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 09:27:58.031569 20752 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: Deleting consensus metadata
I20260809 09:27:58.031752 20752 master_runner.cc:315] Copying system tablet from 127.9.102.62:37729
I20260809 09:27:58.032860 20752 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: tablet copy: Beginning tablet copy session from remote peer at address 127.9.102.62:37729
I20260809 09:27:58.040486 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.033391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57930 (local address 127.9.102.62:37729)
0809 09:27:58.033510 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.033515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.033723 (+ 208us) server_negotiation.cc:407] Connection header received
0809 09:27:58.033857 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.033860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.033900 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.033992 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.034378 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.034875 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.035793 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.036019 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.036173 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.037215 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.037219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.037222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.037244 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.038700 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.039117 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.039120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.039122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.039197 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.039543 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.039548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.039553 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.039740 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.039880 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.040223 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.040336 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 09:27:58.040503 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.033350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:57930)
0809 09:27:58.033635 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.033647 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.033762 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.034006 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.034009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.034191 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.034268 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.034273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.035006 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.035011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.035661 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.035672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.036163 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.036167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.036274 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.036763 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.036778 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.037076 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.038812 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.038817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.038819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.039018 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.039286 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.039290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.039291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.039453 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.039882 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.039885 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.039981 (+ 96us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.040092 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.040310 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":90,"threads_started":1}
I20260809 09:27:58.041216 20042 tablet_copy_service.cc:140] P c1ef11a47c3845b09d02d5a1472c860e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer edc35643011b4a2e8ee9a33dac0243d2 ({username='oryx', principal='oryx/127.9.102.61@KRBTEST.COM'} at 127.0.0.1:57930)
I20260809 09:27:58.041376 20042 tablet_copy_service.cc:161] P c1ef11a47c3845b09d02d5a1472c860e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer edc35643011b4a2e8ee9a33dac0243d2 at {username='oryx', principal='oryx/127.9.102.61@KRBTEST.COM'} at 127.0.0.1:57930: session id = edc35643011b4a2e8ee9a33dac0243d2-00000000000000000000000000000000
I20260809 09:27:58.042995 20042 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 09:27:58.044620 20752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.60.keytab.
Entry for principal oryx/127.9.102.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.60.keytab.
I20260809 09:27:58.046797 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.60:40201
--webserver_interface=127.9.102.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.60.keytab
--principal=oryx/127.9.102.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:37729,127.9.102.61:46537,127.9.102.60:40201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:27:58.047260 20752 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: tablet copy: Starting download of 0 data blocks...
I20260809 09:27:58.047427 20752 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: tablet copy: Starting download of 1 WAL segments...
I20260809 09:27:58.049190 20752 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 09:27:58.050632 20042 tablet_copy_service.cc:342] P c1ef11a47c3845b09d02d5a1472c860e: Request end of tablet copy session edc35643011b4a2e8ee9a33dac0243d2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.102.61@KRBTEST.COM'} at 127.0.0.1:57930
I20260809 09:27:58.050746 20042 tablet_copy_service.cc:433] P c1ef11a47c3845b09d02d5a1472c860e: ending tablet copy session edc35643011b4a2e8ee9a33dac0243d2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer edc35643011b4a2e8ee9a33dac0243d2
I20260809 09:27:58.057415 20752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:58.058868 20927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:58.058899 20931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:58.059433 20928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:58.059497 20752 server_base.cc:1034] running on GCE node
I20260809 09:27:58.059800 20752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:58.060127 20752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:58.061266 20752 hybrid_clock.cc:648] HybridClock initialized: now 1786267678061275 us; error 41 us; skew 500 ppm
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:58.062664 20752 init.cc:377] Logged in from keytab as oryx/127.9.102.61@KRBTEST.COM (short username oryx)
I20260809 09:27:58.063102 20752 webserver.cc:460] Webserver started at http://127.9.102.61:33121/ using document root <none> and password file <none>
I20260809 09:27:58.063280 20752 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:58.063337 20752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:58.064632 20752 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260809 09:27:58.065471 20940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.065743 20752 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260809 09:27:58.065830 20752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/wal
uuid: "edc35643011b4a2e8ee9a33dac0243d2"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "6a6aff1da3ae25f98b5814474d0751b2"
server_key_iv: "b9a8d637f15799f7f9ff59eecac821a8"
server_key_version: "encryptionkey@0"
I20260809 09:27:58.065901 20752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:58.074769 20752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:58.077054 20752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:58.082746 20752 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.61:46537
I20260809 09:27:58.083216 20752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
I20260809 09:27:58.084616 21027 sys_catalog.cc:263] Verifying existing consensus state
I20260809 09:27:58.085553 21027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: Bootstrap starting.
I20260809 09:27:58.088773 21026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.61:46537 every 8 connection(s)
I20260809 09:27:58.094801 21027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 09:27:58.095181 21027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: Bootstrap complete.
I20260809 09:27:58.095655 21027 raft_consensus.cc:348] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true } }
I20260809 09:27:58.095826 21027 raft_consensus.cc:729] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: edc35643011b4a2e8ee9a33dac0243d2, State: Initialized, Role: LEARNER
I20260809 09:27:58.095950 21027 consensus_queue.cc:260] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true } }
I20260809 09:27:58.096434 21027 sys_catalog.cc:564] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:58.098127 21033 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true } } }
I20260809 09:27:58.098284 21033 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: This master's current role is: LEARNER
I20260809 09:27:58.099732 21052 catalog_manager.cc:1259] Loaded cluster ID: 893b9bc4eb0343f08d68c4feac02e2a7
I20260809 09:27:58.099818 21052 catalog_manager.cc:1552] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: loading cluster ID for follower catalog manager: success
I20260809 09:27:58.102025 21052 catalog_manager.cc:1574] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: acquiring CA information for follower catalog manager: success
I20260809 09:27:58.103240 21052 catalog_manager.cc:1602] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 09:27:58.182236 20925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:58.182466 20925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:58.182529 20925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:58.185942 20925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:27:58.186012 20925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:58.186061 20925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:58.186117 20925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:27:58.186151 20925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:27:58.190733 20925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:37729,127.9.102.61:46537,127.9.102.60:40201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.60.keytab
--principal=oryx/127.9.102.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.60:40201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.102.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:58.191608 20925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:58.192555 20925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:58.200376 21064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:58.200410 21062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:58.200395 21060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:58.200600 20925 server_base.cc:1034] running on GCE node
I20260809 09:27:58.201334 20925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:58.202335 20925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:58.203507 20925 hybrid_clock.cc:648] HybridClock initialized: now 1786267678203485 us; error 42 us; skew 500 ppm
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:58.205987 20925 init.cc:377] Logged in from keytab as oryx/127.9.102.60@KRBTEST.COM (short username oryx)
I20260809 09:27:58.206852 20925 webserver.cc:460] Webserver started at http://127.9.102.60:38289/ using document root <none> and password file <none>
I20260809 09:27:58.207345 20925 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:58.207429 20925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:58.207664 20925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:27:58.209602 20925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data/instance:
uuid: "fb31c6aa350c4812a44f6ad842d571cd"
format_stamp: "Formatted at 2026-08-09 09:27:58 on dist-test-slave-7vjv"
server_key: "83080bb865ed742ecb4c5bbec27de511"
server_key_iv: "d781413470ada1c73727f2b2dbcb7fa7"
server_key_version: "encryptionkey@0"
I20260809 09:27:58.210155 20925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/wal/instance:
uuid: "fb31c6aa350c4812a44f6ad842d571cd"
format_stamp: "Formatted at 2026-08-09 09:27:58 on dist-test-slave-7vjv"
server_key: "83080bb865ed742ecb4c5bbec27de511"
server_key_iv: "d781413470ada1c73727f2b2dbcb7fa7"
server_key_version: "encryptionkey@0"
I20260809 09:27:58.213599 20925 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 09:27:58.215664 21080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.216892 20925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 09:27:58.217020 20925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/wal
uuid: "fb31c6aa350c4812a44f6ad842d571cd"
format_stamp: "Formatted at 2026-08-09 09:27:58 on dist-test-slave-7vjv"
server_key: "83080bb865ed742ecb4c5bbec27de511"
server_key_iv: "d781413470ada1c73727f2b2dbcb7fa7"
server_key_version: "encryptionkey@0"
I20260809 09:27:58.217126 20925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:58.241573 20925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:58.244066 20925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:58.244278 20925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:58.250425 20925 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.60:40201
I20260809 09:27:58.250675 21169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.60:40201 every 8 connection(s)
I20260809 09:27:58.251323 20925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
I20260809 09:27:58.253839 21172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:58.257196 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 20925
I20260809 09:27:58.257612 21172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.60@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.102.60@KRBTEST.COM
I20260809 09:27:58.269708 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.259388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.9.102.62:37729)
0809 09:27:58.259513 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.259517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.259826 (+ 309us) server_negotiation.cc:407] Connection header received
0809 09:27:58.260158 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.260161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.260218 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.260315 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.261188 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.261659 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.262770 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.263176 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.263284 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.265925 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.265929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.265932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.265974 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.267670 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.268285 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.268288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.268290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.268371 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.268701 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.268705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.268707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.268877 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.268983 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.269437 (+ 454us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.269553 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 09:27:58.270498 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.258486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.9.102.60:40201)
0809 09:27:58.258831 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.258853 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.258883 (+ 30us) server_negotiation.cc:407] Connection header received
0809 09:27:58.259622 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.259667 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.259955 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.260217 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.260907 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.262004 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.262831 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.263421 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.263521 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.266446 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.266450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.266458 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.266477 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.268375 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.268797 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.268803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.268805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.268882 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.269162 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.269165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.269167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.269397 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.269592 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.269789 (+ 197us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.269913 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":122,"threads_started":1}
I20260809 09:27:58.270566 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.259153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:57936)
0809 09:27:58.259675 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.259721 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.259941 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.260377 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.260382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.260742 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.261056 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.261068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.261795 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.261798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.262637 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.262646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.263338 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.263353 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.263608 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.264288 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.264308 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.265774 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.267780 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.267790 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.267799 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.268149 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.268464 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.268468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.268470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.268605 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.268983 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.268991 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.269214 (+ 223us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.269396 (+ 182us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.269649 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":212,"threads_started":1}
I20260809 09:27:58.271580 21172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.60@KRBTEST.COM for oryx/127.9.102.61@KRBTEST.COM
I20260809 09:27:58.278364 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.271950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:46537 (local address 127.0.0.1:33360)
0809 09:27:58.272072 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.272087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.272170 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.272522 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.272524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.272637 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.272712 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.272716 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.273491 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.273493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.273925 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.273932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.274288 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.274290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.274350 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.274666 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.274676 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.275669 (+ 993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.277363 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.277365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.277366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.277526 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.277742 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.277744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.277745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.277806 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.278081 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.278082 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.278141 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.278207 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.278285 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260809 09:27:58.278493 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.272116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33360 (local address 127.9.102.61:46537)
0809 09:27:58.272340 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.272343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.272353 (+ 10us) server_negotiation.cc:407] Connection header received
0809 09:27:58.272387 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.272389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.272423 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.272492 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.272807 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.273373 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.274013 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.274204 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.274272 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.275784 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.275786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.275788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.275808 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.277262 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.277603 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.277605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.277606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.277671 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.277872 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.277874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.277875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.278004 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.278078 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.278258 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.278338 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260809 09:27:58.279069 21172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.60@KRBTEST.COM for oryx/127.9.102.60@KRBTEST.COM
I20260809 09:27:58.286056 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.279477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:40201 (local address 127.0.0.1:45146)
0809 09:27:58.279549 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.279561 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.279652 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.279864 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.279865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.279979 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.280046 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.280050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.281134 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.281136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.281562 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.281569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.281963 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.281965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.282013 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.282325 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.282335 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.283303 (+ 968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.284764 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.284766 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.284767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.284928 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.285153 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.285155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.285156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.285221 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.285559 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.285561 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.285626 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.285738 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.285897 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260809 09:27:58.286077 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.279606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45146 (local address 127.9.102.60:40201)
0809 09:27:58.279726 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.279729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.279740 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:27:58.279766 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.279768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.279796 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.279856 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.280197 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.281030 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.281653 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.281876 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.281946 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.283405 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.283408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.283409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.283427 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.284665 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.285004 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.285006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.285007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.285077 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.285291 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.285293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.285294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.285466 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.285555 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.285780 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.285953 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":61}
I20260809 09:27:58.288489 21172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: Bootstrap starting.
I20260809 09:27:58.291121 21172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:58.291960 21172 log.cc:824] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:58.293610 21172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: No bootstrap required, opened a new log
I20260809 09:27:58.297397 21172 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } }
I20260809 09:27:58.297616 21172 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:58.297684 21172 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb31c6aa350c4812a44f6ad842d571cd, State: Initialized, Role: FOLLOWER
I20260809 09:27:58.298095 21172 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } }
I20260809 09:27:58.298542 21184 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } } }
I20260809 09:27:58.298658 21184 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:27:58.299180 21172 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: configured and running, proceeding with master startup.
W20260809 09:27:58.304016 21195 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:27:58.304121 21195 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:27:58.307057 20925 master_runner.cc:428] Detected that this master fb31c6aa350c4812a44f6ad842d571cd is joining an existing cluster
I20260809 09:27:58.307111 20925 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.102.60:40201
I20260809 09:27:58.318084 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.309193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.9.102.62:37729)
0809 09:27:58.309287 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.309291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.310012 (+ 721us) server_negotiation.cc:407] Connection header received
0809 09:27:58.310186 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.310189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.310235 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.310279 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.311360 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.312061 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.313365 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.313658 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.313717 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.315297 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.315300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.315302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.315322 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.316629 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.316999 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.317002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.317004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.317087 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.317337 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.317341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.317342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.317492 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.317591 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.317864 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.317957 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 09:27:58.318084 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.309392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:57952)
0809 09:27:58.309860 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.309873 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.310077 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.310418 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.310420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.311045 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.311254 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.311263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.312264 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.312267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.313238 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.313248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.313922 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.313926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.314055 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.314882 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.314892 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.315167 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.316730 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.316734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.316736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.316910 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.317170 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.317174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.317175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.317256 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.317583 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.317586 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.317713 (+ 127us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.317815 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.317930 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":48,"thread_start_us":114,"threads_started":1}
I20260809 09:27:58.323244 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 20117
I20260809 09:27:58.342969 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.310137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33376 (local address 127.9.102.61:46537)
0809 09:27:58.310262 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.310266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.310278 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:27:58.310317 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.310319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.310349 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.310406 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.311324 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.312049 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.313477 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.313755 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.313817 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.335188 (+ 21371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.335193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.335197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.335225 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.340561 (+ 5336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.341718 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.341721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.341723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.341805 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.342149 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.342154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.342157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.342352 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.342442 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.342744 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.342851 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 09:27:58.343012 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.309331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:46537 (local address 127.0.0.1:33376)
0809 09:27:58.309645 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.309667 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.309764 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.310432 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.310434 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.310963 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.311189 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.311198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.312264 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.312267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.313362 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.313370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.313963 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.313965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.314102 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.334511 (+ 20409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.334525 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.335043 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.340693 (+ 5650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.340696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.340698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.340929 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.341929 (+ 1000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.341932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.341934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.342020 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.342486 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.342491 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.342590 (+ 99us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.342696 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.342822 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":243,"thread_start_us":98,"threads_started":1}
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267677, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.62@KRBTEST.COM for oryx/127.9.102.60@KRBTEST.COM
I20260809 09:27:58.348716 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.310239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.9.102.60:40201)
0809 09:27:58.310503 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.310506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.310514 (+ 8us) server_negotiation.cc:407] Connection header received
0809 09:27:58.310544 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.310545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.310569 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.310645 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.311533 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.312340 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.313756 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.314119 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.314220 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.344559 (+ 30339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.344564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.344569 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.344589 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.346658 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.347225 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.347230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.347234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.347312 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.347712 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.347717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.347721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.347961 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.348031 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.348370 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.348505 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":213}
I20260809 09:27:58.349025 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.309744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:40201 (local address 127.0.0.1:45154)
0809 09:27:58.310138 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.310150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.310240 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.310762 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.310765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.311072 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.311414 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.311424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.312542 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.312545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.313644 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.313655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.314317 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.314320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.314396 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.344025 (+ 29629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.344041 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.344451 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.346797 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.346803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.346807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.347109 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.347518 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.347523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.347528 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.347624 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.348110 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.348115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.348208 (+ 93us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.348640 (+ 432us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.348764 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":7,"spinlock_wait_cycles":65792,"thread_start_us":131,"threads_started":1}
I20260809 09:27:58.352694 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.321111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:40201 (local address 127.0.0.1:45162)
0809 09:27:58.321380 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.321394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.321461 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.321691 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.321693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.321857 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.321938 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.321944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.323081 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.323084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.323654 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.323663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.346470 (+ 22807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.346473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.346586 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.347038 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.347054 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.348406 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.350803 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.350809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.350813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.351082 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.351359 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.351365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.351369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.351465 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.352038 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.352044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.352190 (+ 146us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.352308 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.352470 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260809 09:27:58.352701 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.321195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.9.102.60:40201)
0809 09:27:58.321449 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.321455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.321466 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:27:58.321544 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.321546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.321591 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.321653 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.322039 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.322719 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.346112 (+ 23393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.346353 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.346464 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.348584 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.348587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.348590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.348611 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.350682 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.351181 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.351184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.351186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.351264 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.351592 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.351610 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.351612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.351863 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.352016 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.352373 (+ 357us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.352486 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":133,"threads_started":1}
I20260809 09:27:58.353251 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:38807
--local_ip_for_outbound_sockets=127.9.102.1
--tserver_master_addrs=127.9.102.62:37729
--webserver_port=43003
--webserver_interface=127.9.102.1
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab
--principal=oryx/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.102.62:37729,127.9.102.61:46537
--tserver_master_addrs=127.9.102.62:37729,127.9.102.61:46537,127.9.102.60:40201 with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:27:58.353482 20011 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.102.60:40201: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd"
member_type: NON_VOTER
last_known_addr {
host: "127.9.102.60"
port: 40201
}
attrs {
promote: true
}
}
dest_uuid: "c1ef11a47c3845b09d02d5a1472c860e"
cas_config_opid_index: 5
I20260809 09:27:58.354097 20011 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } }
I20260809 09:27:58.355260 20906 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fb31c6aa350c4812a44f6ad842d571cd (127.9.102.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:58.356832 20906 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.102.60:40201
I20260809 09:27:58.357003 20906 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:58.357087 20906 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:58.357537 21126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:58.359580 21126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 FOLLOWER]: Refusing update from remote peer c1ef11a47c3845b09d02d5a1472c860e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 09:27:58.360066 20925 master.cc:537] Master@127.9.102.60:40201 shutting down...
I20260809 09:27:58.360153 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.355662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.9.102.61:46537)
0809 09:27:58.355768 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.355772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.355905 (+ 133us) server_negotiation.cc:407] Connection header received
0809 09:27:58.356042 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.356044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.356092 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.356175 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:58.356539 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.357224 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.358713 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.359311 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.359399 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.359541 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.359934 (+ 393us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.360016 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 09:27:58.360242 20906 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:58.360504 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.355321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:46537 (local address 127.0.0.1:33380)
0809 09:27:58.355847 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.355859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.355954 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.356299 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.356301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.356315 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:58.356423 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.356428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.357365 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.357368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.358565 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.358576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.359669 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.359673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.359755 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.359794 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.360320 (+ 526us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.360400 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476}
I20260809 09:27:58.360960 20983 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 LEARNER]: Refusing update from remote peer c1ef11a47c3845b09d02d5a1472c860e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 09:27:58.361498 20909 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260809 09:27:58.362071 19987 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e -> Peer fb31c6aa350c4812a44f6ad842d571cd (127.9.102.60:40201): Couldn't send request to peer fb31c6aa350c4812a44f6ad842d571cd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 09:27:58.363523 20983 raft_consensus.cc:2932] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fb31c6aa350c4812a44f6ad842d571cd (127.9.102.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:58.365257 21033 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1ef11a47c3845b09d02d5a1472c860e. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:58.365377 21033 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: This master's current role is: LEARNER
I20260809 09:27:58.365473 21033 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: NON_VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: true } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:58.365537 21033 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: This master's current role is: LEARNER
I20260809 09:27:58.365886 20909 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e: attempting to promote NON_VOTER edc35643011b4a2e8ee9a33dac0243d2 to VOTER
I20260809 09:27:58.366264 20909 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } }
W20260809 09:27:58.367839 19987 consensus_peers.cc:487] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e -> Peer fb31c6aa350c4812a44f6ad842d571cd (127.9.102.60:40201): Couldn't send request to peer fb31c6aa350c4812a44f6ad842d571cd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 09:27:58.368046 20983 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 LEARNER]: Refusing update from remote peer c1ef11a47c3845b09d02d5a1472c860e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 09:27:58.368496 21221 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 09:27:58.368966 20925 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:58.369215 20925 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:58.369310 20925 tablet_replica.cc:330] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: stopping tablet replica
I20260809 09:27:58.371510 20906 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, edc35643011b4a2e8ee9a33dac0243d2 (127.9.102.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:58.373390 20906 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:58.373494 20906 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:58.374006 20909 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:58.374114 20909 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:58.374135 20925 master.cc:552] Master@127.9.102.60:40201 shutdown complete.
I20260809 09:27:58.374164 20925 master_runner.cc:305] Clearing existing system tablet
I20260809 09:27:58.374398 20925 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:58.374451 20925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:58.374794 20983 raft_consensus.cc:2932] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, edc35643011b4a2e8ee9a33dac0243d2 (127.9.102.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:58.375564 20925 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260809 09:27:58.376123 21220 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:58.376238 21220 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:27:58.376646 21224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.376823 20925 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:58.376967 20925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 09:27:58.377013 20925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/wal
uuid: "fb31c6aa350c4812a44f6ad842d571cd"
format_stamp: "Formatted at 2026-08-09 09:27:58 on dist-test-slave-7vjv"
server_key: "83080bb865ed742ecb4c5bbec27de511"
server_key_iv: "d781413470ada1c73727f2b2dbcb7fa7"
server_key_version: "encryptionkey@0"
I20260809 09:27:58.378448 20925 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:58.380053 20925 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 09:27:58.380168 20925 log.cc:1196] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 09:27:58.380400 20925 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: Deleting consensus metadata
I20260809 09:27:58.380579 20925 master_runner.cc:315] Copying system tablet from 127.9.102.62:37729
I20260809 09:27:58.381567 20925 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: tablet copy: Beginning tablet copy session from remote peer at address 127.9.102.62:37729
I20260809 09:27:58.388509 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.382119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.9.102.62:37729)
0809 09:27:58.382215 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.382218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.382391 (+ 173us) server_negotiation.cc:407] Connection header received
0809 09:27:58.382494 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.382497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.382533 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.382616 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.383012 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.383469 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.384273 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.384551 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.384644 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.385631 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.385634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.385636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.385658 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.387051 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.387462 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.387466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.387468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.387544 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.387792 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.387794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.387796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.387946 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.388037 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.388299 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.388398 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 09:27:58.388525 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.382086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:57968)
0809 09:27:58.382317 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.382327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.382427 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.382633 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.382635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.382812 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.382906 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.382912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.383572 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.383576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.384086 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.384154 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.384655 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.384659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.384742 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.385204 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.385220 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.385505 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.387154 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.387158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.387160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.387385 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.387624 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.387627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.387629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.387706 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.388041 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.388045 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.388150 (+ 105us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.388253 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.388372 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":56,"threads_started":1}
I20260809 09:27:58.389055 20042 tablet_copy_service.cc:140] P c1ef11a47c3845b09d02d5a1472c860e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fb31c6aa350c4812a44f6ad842d571cd ({username='oryx', principal='oryx/127.9.102.60@KRBTEST.COM'} at 127.0.0.1:57968)
I20260809 09:27:58.389147 20042 tablet_copy_service.cc:161] P c1ef11a47c3845b09d02d5a1472c860e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fb31c6aa350c4812a44f6ad842d571cd at {username='oryx', principal='oryx/127.9.102.60@KRBTEST.COM'} at 127.0.0.1:57968: session id = fb31c6aa350c4812a44f6ad842d571cd-00000000000000000000000000000000
I20260809 09:27:58.389809 20042 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 09:27:58.390790 20925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 09:27:58.392506 20925 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: tablet copy: Starting download of 0 data blocks...
I20260809 09:27:58.392689 20925 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: tablet copy: Starting download of 1 WAL segments...
I20260809 09:27:58.393939 20925 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 09:27:58.395218 20042 tablet_copy_service.cc:342] P c1ef11a47c3845b09d02d5a1472c860e: Request end of tablet copy session fb31c6aa350c4812a44f6ad842d571cd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.102.60@KRBTEST.COM'} at 127.0.0.1:57968
I20260809 09:27:58.395309 20042 tablet_copy_service.cc:433] P c1ef11a47c3845b09d02d5a1472c860e: ending tablet copy session fb31c6aa350c4812a44f6ad842d571cd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fb31c6aa350c4812a44f6ad842d571cd
I20260809 09:27:58.401001 20925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:58.402456 21238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:58.404769 21237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:58.408985 21241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:58.409053 20925 server_base.cc:1034] running on GCE node
I20260809 09:27:58.409399 20925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:58.409723 20925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:58.410879 20925 hybrid_clock.cc:648] HybridClock initialized: now 1786267678410858 us; error 41 us; skew 500 ppm
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:58.412288 20925 init.cc:377] Logged in from keytab as oryx/127.9.102.60@KRBTEST.COM (short username oryx)
I20260809 09:27:58.412750 20925 webserver.cc:460] Webserver started at http://127.9.102.60:41305/ using document root <none> and password file <none>
I20260809 09:27:58.412914 20925 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:58.412963 20925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:58.414099 20925 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260809 09:27:58.415002 21255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.415248 20925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 09:27:58.415336 20925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/wal
uuid: "fb31c6aa350c4812a44f6ad842d571cd"
format_stamp: "Formatted at 2026-08-09 09:27:58 on dist-test-slave-7vjv"
server_key: "83080bb865ed742ecb4c5bbec27de511"
server_key_iv: "d781413470ada1c73727f2b2dbcb7fa7"
server_key_version: "encryptionkey@0"
I20260809 09:27:58.415416 20925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:58.421298 20925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:58.423117 20925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:58.428231 20925 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.60:40201
I20260809 09:27:58.428314 21346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.60:40201 every 8 connection(s)
I20260809 09:27:58.428706 20925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
I20260809 09:27:58.429919 21347 sys_catalog.cc:263] Verifying existing consensus state
I20260809 09:27:58.430648 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: Bootstrap starting.
I20260809 09:27:58.438683 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 09:27:58.438988 21347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: Bootstrap complete.
I20260809 09:27:58.439400 21347 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } }
I20260809 09:27:58.439544 21347 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb31c6aa350c4812a44f6ad842d571cd, State: Initialized, Role: LEARNER
I20260809 09:27:58.439683 21347 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } }
I20260809 09:27:58.440009 21352 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:58.440188 21347 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:27:58.440269 21352 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: This master's current role is: LEARNER
I20260809 09:27:58.442878 21373 catalog_manager.cc:1259] Loaded cluster ID: 893b9bc4eb0343f08d68c4feac02e2a7
I20260809 09:27:58.442987 21373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: loading cluster ID for follower catalog manager: success
I20260809 09:27:58.444625 21373 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: acquiring CA information for follower catalog manager: success
I20260809 09:27:58.445704 21373 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 09:27:58.507447 21215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:58.507758 21215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:58.507808 21215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:58.514168 21215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:58.514232 21215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:58.514307 21215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:27:58.521103 21215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab
--principal=oryx/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:38807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=43003
--tserver_master_addrs=127.9.102.62:37729,127.9.102.61:46537,127.9.102.60:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:58.522280 21215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:58.523339 21215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:58.532387 21382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:58.532434 21380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:58.532434 21379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:58.532635 21215 server_base.cc:1034] running on GCE node
I20260809 09:27:58.533159 21215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:58.533697 21215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:58.534885 21215 hybrid_clock.cc:648] HybridClock initialized: now 1786267678534858 us; error 35 us; skew 500 ppm
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:58.538146 21215 init.cc:377] Logged in from keytab as oryx/127.9.102.1@KRBTEST.COM (short username oryx)
I20260809 09:27:58.538815 21215 webserver.cc:460] Webserver started at http://127.9.102.1:43003/ using document root <none> and password file <none>
I20260809 09:27:58.539264 21215 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:58.539327 21215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:58.543239 21215 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 09:27:58.545037 21395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.546082 21215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:58.546165 21215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "060f8bc83055420a98a44897a75f6fb1"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "97a6f49ed0fab4f8ff931cdcd0de30e1"
server_key_iv: "31841f7d57aed73ad71abb6e949c9327"
server_key_version: "encryptionkey@0"
I20260809 09:27:58.546576 21215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:58.553162 20539 heartbeater.cc:503] Master 127.9.102.62:37729 was elected leader, sending a full tablet report...
I20260809 09:27:58.568399 21215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:58.570798 21215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:58.570976 21215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:58.571507 21215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:58.572429 21215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:58.572481 21215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.572516 21215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:58.572531 21215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.584367 21215 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:38807
I20260809 09:27:58.584389 21579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:38807 every 8 connection(s)
I20260809 09:27:58.585342 21215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:27:58.587214 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 21215
I20260809 09:27:58.587423 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 20334
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.61@KRBTEST.COM
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.60@KRBTEST.COM
I20260809 09:27:58.601931 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:34507
--local_ip_for_outbound_sockets=127.9.102.2
--tserver_master_addrs=127.9.102.62:37729
--webserver_port=38339
--webserver_interface=127.9.102.2
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab
--principal=oryx/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.102.62:37729,127.9.102.61:46537
--tserver_master_addrs=127.9.102.62:37729,127.9.102.61:46537,127.9.102.60:40201 with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:27:58.605450 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.586986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:59961 (local address 127.9.102.61:46537)
0809 09:27:58.587109 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.587115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.591142 (+ 4027us) server_negotiation.cc:407] Connection header received
0809 09:27:58.592455 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.592461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.592521 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.592573 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.594123 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.594889 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.596896 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.597340 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.597458 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.601097 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.601103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.601108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.601137 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.602951 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.603610 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.603614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.603615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.603701 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.604156 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.604160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.604162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.604401 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.604491 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.605167 (+ 676us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.605280 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 09:27:58.606348 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.588491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:40201 (local address 127.9.102.1:55439)
0809 09:27:58.594949 (+ 6458us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.594961 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.595077 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.595457 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.595462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.595746 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.595971 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.595980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.597458 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.597461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.598407 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.598417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.599225 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.599231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.599364 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.599932 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.599949 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.601491 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.603489 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.603493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.603497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.603765 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.604108 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.604112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.604113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.604207 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.604678 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.604683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.604870 (+ 187us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.605066 (+ 196us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.605373 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6395,"mutex_wait_us":6,"spinlock_wait_cycles":8192,"thread_start_us":1929,"threads_started":1}
I20260809 09:27:58.606943 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.587907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:51137 (local address 127.9.102.62:37729)
0809 09:27:58.588027 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.588032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.595863 (+ 7831us) server_negotiation.cc:407] Connection header received
0809 09:27:58.596083 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.596089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.596165 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.596227 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.596921 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.597689 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.599300 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.599644 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.599983 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.603594 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.603597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.603600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.603624 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.605225 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.605697 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.605702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.605704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.605781 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.606058 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.606061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.606063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.606240 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.606302 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.606726 (+ 424us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.606822 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 09:27:58.607194 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.587523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.1:51137)
0809 09:27:58.595753 (+ 8230us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.595766 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.595945 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.596366 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.596369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.596635 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.596762 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.596770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.598031 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.598035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.599153 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.599165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.600163 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.600169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.600425 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.600902 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.600914 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.603429 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.605340 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.605343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.605345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.605565 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.605876 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.605881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.605883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.605958 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.606372 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.606377 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.606465 (+ 88us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.606673 (+ 208us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.606792 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8147,"mutex_wait_us":3031,"thread_start_us":163,"threads_started":1}
I20260809 09:27:58.607210 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.590970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:55439 (local address 127.9.102.60:40201)
0809 09:27:58.592847 (+ 1877us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.592851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.595053 (+ 2202us) server_negotiation.cc:407] Connection header received
0809 09:27:58.595175 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.595180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.595239 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.595318 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.596164 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.597312 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.598522 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.598959 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.599089 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.601666 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.601669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.601672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.601697 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.603358 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.603878 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.603883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.603886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.603974 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.604312 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.604316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.604318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.604538 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.604627 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.606851 (+ 2224us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.606985 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1805,"thread_start_us":1681,"threads_started":1}
I20260809 09:27:58.606345 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.587523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:46537 (local address 127.9.102.1:59961)
0809 09:27:58.590988 (+ 3465us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.591055 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.592228 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.592953 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.592963 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.593404 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.593958 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.593975 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.595052 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.595058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.596745 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.596760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.597654 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.597660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.597930 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.598822 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.598841 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.600872 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.603158 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.603163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.603175 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.603512 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.603819 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.603826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.603829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.604003 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.604565 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.604571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.604872 (+ 301us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.605889 (+ 1017us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.606016 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3273,"mutex_wait_us":52,"spinlock_wait_cycles":6656,"thread_start_us":126,"threads_started":1}
I20260809 09:27:58.608269 21581 heartbeater.cc:347] Connected to a master server at 127.9.102.62:37729
I20260809 09:27:58.608312 21580 heartbeater.cc:347] Connected to a master server at 127.9.102.60:40201
I20260809 09:27:58.608479 21581 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:58.608479 21580 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:58.608629 21582 heartbeater.cc:347] Connected to a master server at 127.9.102.61:46537
I20260809 09:27:58.608718 21582 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:58.609009 21582 heartbeater.cc:511] Master 127.9.102.61:46537 requested a full tablet report, sending...
I20260809 09:27:58.609045 21580 heartbeater.cc:511] Master 127.9.102.60:40201 requested a full tablet report, sending...
I20260809 09:27:58.609009 21581 heartbeater.cc:511] Master 127.9.102.62:37729 requested a full tablet report, sending...
I20260809 09:27:58.610587 20962 ts_manager.cc:194] Registered new tserver with Master: 060f8bc83055420a98a44897a75f6fb1 (127.9.102.1:38807)
I20260809 09:27:58.610569 21287 ts_manager.cc:194] Registered new tserver with Master: 060f8bc83055420a98a44897a75f6fb1 (127.9.102.1:38807)
I20260809 09:27:58.611301 20011 ts_manager.cc:194] Re-registered known tserver with Master: 060f8bc83055420a98a44897a75f6fb1 (127.9.102.1:38807)
I20260809 09:27:58.611800 20011 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:51137
W20260809 09:27:58.705797 21597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:58.706082 21597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:58.706149 21597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:58.709721 21597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:58.709787 21597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:58.709858 21597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:27:58.714380 21597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab
--principal=oryx/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:34507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=38339
--tserver_master_addrs=127.9.102.62:37729,127.9.102.61:46537,127.9.102.60:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:58.715169 21597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:58.715996 21597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:58.723889 21607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:58.723918 21597 server_base.cc:1034] running on GCE node
W20260809 09:27:58.723953 21603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:58.723891 21611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:58.724502 21597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:58.725041 21597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:58.726215 21597 hybrid_clock.cc:648] HybridClock initialized: now 1786267678726202 us; error 44 us; skew 500 ppm
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:58.728859 21597 init.cc:377] Logged in from keytab as oryx/127.9.102.2@KRBTEST.COM (short username oryx)
I20260809 09:27:58.729650 21597 webserver.cc:460] Webserver started at http://127.9.102.2:38339/ using document root <none> and password file <none>
I20260809 09:27:58.730111 21597 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:58.730211 21597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:58.733445 21597 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:27:58.735561 21625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.736788 21597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:27:58.736896 21597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "84f801fb831045dab3d877042086156a"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "97eabb0684f33aff9243fabec0832321"
server_key_iv: "4090936cec3527bb0b4e10114cdcc1f9"
server_key_version: "encryptionkey@0"
I20260809 09:27:58.737310 21597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:58.746049 21597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:58.748306 21597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:58.748483 21597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:58.748971 21597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:58.749717 21597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:58.749763 21597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.749822 21597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:58.749851 21597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.750844 20746 heartbeater.cc:503] Master 127.9.102.62:37729 was elected leader, sending a full tablet report...
I20260809 09:27:58.758057 21597 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:34507
I20260809 09:27:58.758122 21822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:34507 every 8 connection(s)
I20260809 09:27:58.758864 21597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:27:58.768386 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 21597
I20260809 09:27:58.768563 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 20547
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.60@KRBTEST.COM
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.61@KRBTEST.COM
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:27:58.787745 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.762296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:43361 (local address 127.9.102.60:40201)
0809 09:27:58.762426 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.762429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.762443 (+ 14us) server_negotiation.cc:407] Connection header received
0809 09:27:58.762904 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.762908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.762969 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.763061 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.764426 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.765165 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.766654 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.766943 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.767024 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.783465 (+ 16441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.783471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.783477 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.783510 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.785361 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.786003 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.786007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.786008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.786096 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.786525 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.786528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.786530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.786732 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.786802 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.787479 (+ 677us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.787602 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:27:58.787745 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.760595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:33483 (local address 127.9.102.61:46537)
0809 09:27:58.760701 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.760704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.761751 (+ 1047us) server_negotiation.cc:407] Connection header received
0809 09:27:58.763003 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.763008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.763062 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.763126 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.764456 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.765038 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.767104 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.767285 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.767381 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.783418 (+ 16037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.783422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.783425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.783453 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.785224 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.786003 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.786007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.786008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.786096 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.786532 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.786536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.786537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.786741 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.786802 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.787479 (+ 677us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.787601 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 09:27:58.787976 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.760508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:52537 (local address 127.9.102.62:37729)
0809 09:27:58.760640 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.760644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.761749 (+ 1105us) server_negotiation.cc:407] Connection header received
0809 09:27:58.762928 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.762932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.762985 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.763048 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.764464 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.765165 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.766360 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.766690 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.766782 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.783757 (+ 16975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.783766 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.783768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.783790 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.786027 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.786552 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.786558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.786562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.786647 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.786947 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.786952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.786956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.787180 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.787247 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.787694 (+ 447us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.787820 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 09:27:58.788633 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:41209
--local_ip_for_outbound_sockets=127.9.102.3
--tserver_master_addrs=127.9.102.62:37729
--webserver_port=33367
--webserver_interface=127.9.102.3
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab
--principal=oryx/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.102.62:37729,127.9.102.61:46537
--tserver_master_addrs=127.9.102.62:37729,127.9.102.61:46537,127.9.102.60:40201 with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:27:58.789166 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.760976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.2:52537)
0809 09:27:58.761606 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.761660 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.762716 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.763261 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.763268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.763715 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.764307 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.764320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.765261 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.765264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.766234 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.766243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.766832 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.766837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.767035 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.767919 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.767930 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.783586 (+ 15656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.786147 (+ 2561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.786152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.786157 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.786445 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.786752 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.786757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.786762 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.786845 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.787320 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.787325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.787425 (+ 100us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.788624 (+ 1199us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.788770 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":152,"spinlock_wait_cycles":128896,"thread_start_us":222,"threads_started":1}
I20260809 09:27:58.789234 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.760976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:46537 (local address 127.9.102.2:33483)
0809 09:27:58.761606 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.761660 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.762716 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.763261 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.763268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.763703 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.764305 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.764320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.765173 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.765175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.766537 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.766543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.767503 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.767508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.767635 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.768124 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.768136 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.783223 (+ 15087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.785357 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.785362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.785375 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.785888 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.786208 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.786210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.786212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.786427 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.786885 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.786890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.787212 (+ 322us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.787906 (+ 694us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.788331 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":284,"spinlock_wait_cycles":116864,"thread_start_us":207,"threads_started":1}
I20260809 09:27:58.790295 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.762052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:40201 (local address 127.9.102.2:43361)
0809 09:27:58.762321 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.762334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.762727 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.763260 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.763265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.763734 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.764302 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.764318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.765494 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.765498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.766532 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.766543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.767052 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.767055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.767204 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.767757 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.767775 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.783226 (+ 15451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.785466 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.785470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.785472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.785888 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.786208 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.786210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.786212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.786415 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.786885 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.786890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.787212 (+ 322us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.787883 (+ 671us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.789845 (+ 1962us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":140,"spinlock_wait_cycles":13056,"thread_start_us":89,"threads_started":1}
I20260809 09:27:58.790381 21826 heartbeater.cc:347] Connected to a master server at 127.9.102.62:37729
I20260809 09:27:58.790446 21827 heartbeater.cc:347] Connected to a master server at 127.9.102.61:46537
I20260809 09:27:58.790576 21826 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:58.790576 21827 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:58.791144 21827 heartbeater.cc:511] Master 127.9.102.61:46537 requested a full tablet report, sending...
I20260809 09:27:58.791145 21826 heartbeater.cc:511] Master 127.9.102.62:37729 requested a full tablet report, sending...
I20260809 09:27:58.791882 21824 heartbeater.cc:347] Connected to a master server at 127.9.102.60:40201
I20260809 09:27:58.791991 21824 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:58.792157 21824 heartbeater.cc:511] Master 127.9.102.60:40201 requested a full tablet report, sending...
I20260809 09:27:58.792197 20011 ts_manager.cc:194] Re-registered known tserver with Master: 84f801fb831045dab3d877042086156a (127.9.102.2:34507)
I20260809 09:27:58.792805 21287 ts_manager.cc:194] Registered new tserver with Master: 84f801fb831045dab3d877042086156a (127.9.102.2:34507)
I20260809 09:27:58.792968 20011 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:52537
I20260809 09:27:58.793689 20962 ts_manager.cc:194] Registered new tserver with Master: 84f801fb831045dab3d877042086156a (127.9.102.2:34507)
I20260809 09:27:58.876224 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.872765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:40201 (local address 127.0.0.1:45166)
0809 09:27:58.873008 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.873040 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.873178 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.873464 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.873467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.873486 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:58.873597 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.873602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.874317 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.874321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.875155 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.875165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.875808 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.875812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.875893 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.875915 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.875993 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.876074 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260809 09:27:58.876222 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.872851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.9.102.60:40201)
0809 09:27:58.872999 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.873003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.873118 (+ 115us) server_negotiation.cc:407] Connection header received
0809 09:27:58.873297 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.873304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.873356 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.873449 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:58.873704 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.874185 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.875281 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.875686 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.875803 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.875943 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.876008 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.876073 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 09:27:58.877059 21309 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260809 09:27:58.877497 21352 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1ef11a47c3845b09d02d5a1472c860e. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:58.877647 21352 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: This master's current role is: LEARNER
W20260809 09:27:58.891426 21839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:27:58.891646 21839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:27:58.891739 21839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:27:58.895344 21839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:27:58.895417 21839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:27:58.895502 21839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:27:58.900398 21839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab
--principal=oryx/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:41209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=33367
--tserver_master_addrs=127.9.102.62:37729,127.9.102.61:46537,127.9.102.60:40201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:27:58.901232 21839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:27:58.902076 21839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:27:58.910553 21845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:27:58.910658 21848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:58.910694 21839 server_base.cc:1034] running on GCE node
W20260809 09:27:58.910629 21846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:27:58.911188 21839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:27:58.911682 21839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:27:58.912863 21839 hybrid_clock.cc:648] HybridClock initialized: now 1786267678912850 us; error 31 us; skew 500 ppm
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:27:58.915570 21839 init.cc:377] Logged in from keytab as oryx/127.9.102.3@KRBTEST.COM (short username oryx)
I20260809 09:27:58.916503 21839 webserver.cc:460] Webserver started at http://127.9.102.3:33367/ using document root <none> and password file <none>
I20260809 09:27:58.916985 21839 fs_manager.cc:359] Metadata directory not provided
I20260809 09:27:58.917084 21839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:27:58.920509 21839 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:27:58.922358 21864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.923357 21839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:27:58.923477 21839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "1228037bffae4bb1be226e7a9c103953"
format_stamp: "Formatted at 2026-08-09 09:27:57 on dist-test-slave-7vjv"
server_key: "74963e99f46f062819fda6a3805f15cd"
server_key_iv: "0da7d212a286f4e3f1bb1efbc3439ce5"
server_key_version: "encryptionkey@0"
I20260809 09:27:58.923924 21839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:27:58.956007 21839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:27:58.958391 21839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:27:58.958567 21839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:27:58.959062 21839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:27:58.959864 21839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:27:58.959944 21839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.960011 21839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:27:58.960067 21839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:27:58.968854 21839 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:41209
I20260809 09:27:58.968874 22053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:41209 every 8 connection(s)
I20260809 09:27:58.969692 21839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:27:58.975987 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 21839
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.61@KRBTEST.COM
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
Aug 09 09:27:58 dist-test-slave-7vjv krb5kdc[19941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267678, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.60@KRBTEST.COM
I20260809 09:27:58.986549 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.972442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:38565 (local address 127.9.102.60:40201)
0809 09:27:58.972568 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.972572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.972807 (+ 235us) server_negotiation.cc:407] Connection header received
0809 09:27:58.973723 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.973725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.973794 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.973861 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.975869 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.976718 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.979640 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.979934 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.980054 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.982879 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.982882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.982885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.982908 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.984497 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.984999 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.985003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.985004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.985100 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.985407 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.985413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.985417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.985631 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.985719 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.986234 (+ 515us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.986360 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:27:58.987185 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.971874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.3:58061)
0809 09:27:58.972335 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.972387 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.973419 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.973934 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.973943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.974414 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.974986 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.975003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.977687 (+ 2684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.977690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.978571 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.978585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.979321 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.979325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.979504 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.980013 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.980041 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.981806 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.984221 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.984225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.984237 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.984518 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.984842 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.984844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.984846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.985004 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.985459 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.985465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.985764 (+ 299us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.985964 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.986333 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":179,"spinlock_wait_cycles":63872,"thread_start_us":116,"threads_started":1}
I20260809 09:27:58.987872 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.971268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:58061 (local address 127.9.102.62:37729)
0809 09:27:58.971396 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.971405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.972477 (+ 1072us) server_negotiation.cc:407] Connection header received
0809 09:27:58.973632 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.973636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.973694 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.973769 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.976865 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.977583 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.978693 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.979019 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.979122 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.981993 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.981997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.982001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.982023 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.984089 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.984655 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.984658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.984659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.984750 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.985112 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.985116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.985119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.985320 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.985403 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.987555 (+ 2152us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.987729 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 09:27:58.988214 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.972319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:40201 (local address 127.9.102.3:38565)
0809 09:27:58.972705 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.972717 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.973419 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.973959 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.973961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.974414 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.974986 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.975003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.978721 (+ 3718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.978724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.979478 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.979486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.980195 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.980198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.980290 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.981061 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.981075 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.982745 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.984623 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.984627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.984631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.984884 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.985208 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.985212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.985214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.985300 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.985765 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.985769 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.985924 (+ 155us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.986029 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.987784 (+ 1755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":1620,"spinlock_wait_cycles":26240,"thread_start_us":144,"threads_started":1}
I20260809 09:27:58.989136 22054 heartbeater.cc:347] Connected to a master server at 127.9.102.60:40201
I20260809 09:27:58.989642 22054 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:58.989802 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.977694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.9.102.62:37729)
0809 09:27:58.978304 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.978308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.979868 (+ 1560us) server_negotiation.cc:407] Connection header received
0809 09:27:58.980056 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.980059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.980106 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.980186 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.981854 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.982646 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.983843 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.984185 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.984261 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.986044 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.986047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.986049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.986087 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.987859 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.988390 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.988394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.988396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.988475 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.988811 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.988814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.988815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.989010 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.989113 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.989507 (+ 394us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.989647 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":58,"threads_started":1}
I20260809 09:27:58.990155 22054 heartbeater.cc:511] Master 127.9.102.60:40201 requested a full tablet report, sending...
I20260809 09:27:58.989137 22058 heartbeater.cc:347] Connected to a master server at 127.9.102.62:37729
I20260809 09:27:58.990715 22058 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:58.990863 22058 heartbeater.cc:511] Master 127.9.102.62:37729 requested a full tablet report, sending...
I20260809 09:27:58.991235 21287 ts_manager.cc:194] Registered new tserver with Master: 1228037bffae4bb1be226e7a9c103953 (127.9.102.3:41209)
I20260809 09:27:58.992333 20011 ts_manager.cc:194] Re-registered known tserver with Master: 1228037bffae4bb1be226e7a9c103953 (127.9.102.3:41209)
I20260809 09:27:58.993100 20011 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:58061
I20260809 09:27:58.994489 22071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.972849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:46537 (local address 127.9.102.3:47145)
0809 09:27:58.973163 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:58.973176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:58.973423 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:58.973933 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:58.973940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:58.974394 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:27:58.974998 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.975006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.975931 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.975934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.977403 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:58.977413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.978204 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.978207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:58.978756 (+ 549us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.979331 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:27:58.979349 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:27:58.988779 (+ 9430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:27:58.993059 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.993063 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.993064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.993292 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.993578 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:27:58.993580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:27:58.993582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:27:58.993654 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:27:58.994056 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:27:58.994058 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:27:58.994135 (+ 77us) client_negotiation.cc:769] Sending connection context
0809 09:27:58.994218 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 09:27:58.994337 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":99,"thread_start_us":133,"threads_started":1}
I20260809 09:27:58.994964 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.972993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:47145 (local address 127.9.102.61:46537)
0809 09:27:58.973146 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.973150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.973249 (+ 99us) server_negotiation.cc:407] Connection header received
0809 09:27:58.973685 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.973688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.973737 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.973783 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.975129 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.975654 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.977554 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.977867 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.977935 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.988922 (+ 10987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.988926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.988929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.988955 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.992918 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.993406 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.993409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.993411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.993483 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.993745 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.993747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.993749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.993924 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.994005 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.994712 (+ 707us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.994834 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 09:27:58.995589 22059 heartbeater.cc:347] Connected to a master server at 127.9.102.61:46537
I20260809 09:27:58.995671 22059 heartbeater.cc:464] Registering TS with master...
I20260809 09:27:58.995873 22059 heartbeater.cc:511] Master 127.9.102.61:46537 requested a full tablet report, sending...
I20260809 09:27:58.992983 20009 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 09:27:58.996652 20962 ts_manager.cc:194] Registered new tserver with Master: 1228037bffae4bb1be226e7a9c103953 (127.9.102.3:41209)
W20260809 09:27:58.997232 20009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:27:58.997591 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.978543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.9.102.60:40201)
0809 09:27:58.979297 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.979301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.981641 (+ 2340us) server_negotiation.cc:407] Connection header received
0809 09:27:58.981779 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.981782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.981827 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.981913 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.982898 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.983637 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.984772 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.985097 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.985183 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.987165 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.987170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.987173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.987199 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.995791 (+ 8592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.996273 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.996276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.996278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.996373 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.996654 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.996657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.996659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.996855 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.996970 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.997284 (+ 314us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.997421 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":91,"threads_started":1}
I20260809 09:27:58.998435 22086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:58.978500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.9.102.61:46537)
0809 09:27:58.979243 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:58.979249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:58.989237 (+ 9988us) server_negotiation.cc:407] Connection header received
0809 09:27:58.989268 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:58.989270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:58.989305 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:58.989412 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:58.990185 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.990831 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.991778 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:58.992175 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:58.992271 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:58.993904 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:58.993908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:58.993910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:58.993934 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:58.996544 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.997127 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.997131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.997133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.997224 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:58.997632 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:58.997635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:58.997636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:58.997854 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:58.997963 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:58.998187 (+ 224us) server_negotiation.cc:299] Negotiation successful
0809 09:27:58.998308 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":74,"threads_started":1}
I20260809 09:27:59.000411 21222 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: NON_VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: true } } }
I20260809 09:27:59.000535 21222 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:59.003273 21222 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e: attempting to promote NON_VOTER fb31c6aa350c4812a44f6ad842d571cd to VOTER
I20260809 09:27:59.003576 21222 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } }
I20260809 09:27:59.005347 21307 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 LEARNER]: Refusing update from remote peer c1ef11a47c3845b09d02d5a1472c860e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260809 09:27:59.005684 22090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 09:27:59.005752 20983 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 FOLLOWER]: Refusing update from remote peer c1ef11a47c3845b09d02d5a1472c860e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260809 09:27:59.006855 22090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 09:27:59.007401 22091 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fb31c6aa350c4812a44f6ad842d571cd (127.9.102.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } } }
I20260809 09:27:59.007870 21309 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fb31c6aa350c4812a44f6ad842d571cd (127.9.102.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } } }
I20260809 09:27:59.009271 22090 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } } }
I20260809 09:27:59.009367 22090 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:59.009534 22091 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } } }
I20260809 09:27:59.009646 22091 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:59.009698 22090 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } } }
I20260809 09:27:59.009780 22090 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e [sys.catalog]: This master's current role is: LEADER
I20260809 09:27:59.009795 20983 raft_consensus.cc:2932] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fb31c6aa350c4812a44f6ad842d571cd (127.9.102.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } } }
I20260809 09:27:59.010295 21352 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } } }
I20260809 09:27:59.010404 21352 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:27:59.011464 22102 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1ef11a47c3845b09d02d5a1472c860e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } } }
I20260809 09:27:59.011624 22102 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:27:59.017795 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.012983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38807 (local address 127.0.0.1:44814)
0809 09:27:59.013132 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:59.013146 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:59.013216 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:59.014056 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:59.014074 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:59.014088 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:59.014172 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:59.014177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.015477 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.015481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:59.016631 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:59.016640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.017451 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.017455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:59.017530 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.017561 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 09:27:59.017617 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 09:27:59.017695 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 09:27:59.017920 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.013138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.9.102.1:38807)
0809 09:27:59.013546 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:59.013559 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:59.013586 (+ 27us) server_negotiation.cc:407] Connection header received
0809 09:27:59.013689 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:59.013696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:59.013914 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:59.014105 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:59.014259 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.015349 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.016737 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.017329 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.017453 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.017591 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:59.017650 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 09:27:59.017740 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":188,"threads_started":1}
I20260809 09:27:59.018945 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.013315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34507 (local address 127.0.0.1:41196)
0809 09:27:59.013865 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:59.013876 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:59.013950 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:59.014939 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:59.014944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:59.014966 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:59.015064 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:59.015071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.016523 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.016525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:59.017802 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:59.017813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.018619 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.018622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:59.018701 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.018723 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 09:27:59.018778 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 09:27:59.018837 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":96,"thread_start_us":103,"threads_started":1}
I20260809 09:27:59.019295 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.013648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:41209 (local address 127.0.0.1:44074)
0809 09:27:59.014480 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:59.014490 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:59.014603 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:59.015144 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:59.015147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:59.015165 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:59.015234 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:59.015238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.016486 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.016489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:59.017803 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:59.017813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.018954 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.018957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:59.019044 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.019067 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 09:27:59.019129 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 09:27:59.019190 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":144,"thread_start_us":107,"threads_started":1}
I20260809 09:27:59.019356 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.013847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.9.102.3:41209)
0809 09:27:59.014307 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:59.014318 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:59.014579 (+ 261us) server_negotiation.cc:407] Connection header received
0809 09:27:59.014700 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:59.014704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:59.014845 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:59.015024 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:59.015332 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.016349 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.018147 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.018838 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.018943 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.019053 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:59.019152 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 09:27:59.019229 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":131,"threads_started":1}
I20260809 09:27:59.020653 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.014113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.9.102.2:34507)
0809 09:27:59.014452 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:59.014499 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:59.014522 (+ 23us) server_negotiation.cc:407] Connection header received
0809 09:27:59.014595 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:59.014599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:59.014761 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:59.014937 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:59.015177 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.016443 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.017940 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.018497 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.018608 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.020385 (+ 1777us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:59.020434 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 09:27:59.020512 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":140,"threads_started":1}
I20260809 09:27:59.020947 21470 tablet_service.cc:1467] Processing CreateTablet for tablet 767467d34aed4160a1d075eb8a272722 (DEFAULT_TABLE table=test-table [id=88a3a3db16d844409f2e311d1e2c74a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:59.021858 21470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767467d34aed4160a1d075eb8a272722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:59.022610 21943 tablet_service.cc:1467] Processing CreateTablet for tablet 767467d34aed4160a1d075eb8a272722 (DEFAULT_TABLE table=test-table [id=88a3a3db16d844409f2e311d1e2c74a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:59.023195 21707 tablet_service.cc:1467] Processing CreateTablet for tablet 767467d34aed4160a1d075eb8a272722 (DEFAULT_TABLE table=test-table [id=88a3a3db16d844409f2e311d1e2c74a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:27:59.023579 21943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767467d34aed4160a1d075eb8a272722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:59.024062 21707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767467d34aed4160a1d075eb8a272722. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:27:59.029115 22112 tablet_bootstrap.cc:492] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1: Bootstrap starting.
I20260809 09:27:59.031446 22116 tablet_bootstrap.cc:492] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a: Bootstrap starting.
I20260809 09:27:59.031819 22112 tablet_bootstrap.cc:654] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:59.032410 22114 tablet_bootstrap.cc:492] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953: Bootstrap starting.
I20260809 09:27:59.033027 22112 log.cc:824] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:59.033954 22116 tablet_bootstrap.cc:654] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:59.034382 22114 tablet_bootstrap.cc:654] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953: Neither blocks nor log segments found. Creating new log.
I20260809 09:27:59.034929 22116 log.cc:824] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:59.035197 22112 tablet_bootstrap.cc:492] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1: No bootstrap required, opened a new log
I20260809 09:27:59.035240 22114 log.cc:824] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953: Log is configured to *not* fsync() on all Append() calls
I20260809 09:27:59.035426 22112 ts_tablet_manager.cc:1397] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 09:27:59.038132 22116 tablet_bootstrap.cc:492] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a: No bootstrap required, opened a new log
I20260809 09:27:59.038308 22116 ts_tablet_manager.cc:1397] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260809 09:27:59.038450 22114 tablet_bootstrap.cc:492] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953: No bootstrap required, opened a new log
I20260809 09:27:59.038710 22114 ts_tablet_manager.cc:1397] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 09:27:59.040586 22112 raft_consensus.cc:348] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } }
I20260809 09:27:59.040910 22112 raft_consensus.cc:374] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:59.040966 22112 raft_consensus.cc:729] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060f8bc83055420a98a44897a75f6fb1, State: Initialized, Role: FOLLOWER
I20260809 09:27:59.041388 22112 consensus_queue.cc:260] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } }
I20260809 09:27:59.042346 22112 ts_tablet_manager.cc:1428] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260809 09:27:59.042620 21581 heartbeater.cc:503] Master 127.9.102.62:37729 was elected leader, sending a full tablet report...
I20260809 09:27:59.043845 22116 raft_consensus.cc:348] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } }
I20260809 09:27:59.044071 22116 raft_consensus.cc:374] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:59.044147 22116 raft_consensus.cc:729] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f801fb831045dab3d877042086156a, State: Initialized, Role: FOLLOWER
I20260809 09:27:59.044332 22114 raft_consensus.cc:348] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } }
I20260809 09:27:59.044574 22116 consensus_queue.cc:260] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } }
I20260809 09:27:59.044553 22114 raft_consensus.cc:374] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:27:59.044742 22114 raft_consensus.cc:729] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1228037bffae4bb1be226e7a9c103953, State: Initialized, Role: FOLLOWER
I20260809 09:27:59.045185 22114 consensus_queue.cc:260] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } }
I20260809 09:27:59.045497 22116 ts_tablet_manager.cc:1428] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260809 09:27:59.046092 22114 ts_tablet_manager.cc:1428] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260809 09:27:59.046108 21826 heartbeater.cc:503] Master 127.9.102.62:37729 was elected leader, sending a full tablet report...
I20260809 09:27:59.046505 22126 raft_consensus.cc:482] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:27:59.046604 22126 raft_consensus.cc:504] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } }
I20260809 09:27:59.047158 22058 heartbeater.cc:503] Master 127.9.102.62:37729 was elected leader, sending a full tablet report...
I20260809 09:27:59.049103 22126 leader_election.cc:288] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060f8bc83055420a98a44897a75f6fb1 (127.9.102.1:38807), 1228037bffae4bb1be226e7a9c103953 (127.9.102.3:41209)
I20260809 09:27:59.052392 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.048106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:56807 (local address 127.9.102.1:38807)
0809 09:27:59.048213 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:59.048216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:59.048227 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:27:59.048263 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:59.048265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:59.048306 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:59.048371 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:59.048798 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.049444 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.051259 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.051759 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.051866 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.051937 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:59.052226 (+ 289us) server_negotiation.cc:299] Negotiation successful
0809 09:27:59.052292 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 09:27:59.052496 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.047675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38807 (local address 127.9.102.2:56807)
0809 09:27:59.047879 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:59.047892 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:59.047960 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:59.048387 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:59.048389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:59.048410 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:59.048650 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:59.048661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.049562 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.049564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:59.050729 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:59.050740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.051904 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.051907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:59.052022 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.052054 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 09:27:59.052206 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 09:27:59.052272 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260809 09:27:59.052944 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.048733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:41209 (local address 127.9.102.2:35035)
0809 09:27:59.048923 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 09:27:59.048935 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:27:59.048992 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:27:59.049461 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:27:59.049463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:27:59.049475 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:27:59.049632 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:59.049642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.050496 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.050501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:59.051495 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:27:59.051505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.052515 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.052520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:27:59.052660 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.052695 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 09:27:59.052769 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 09:27:59.052829 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":2}
I20260809 09:27:59.053144 21502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767467d34aed4160a1d075eb8a272722" candidate_uuid: "84f801fb831045dab3d877042086156a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060f8bc83055420a98a44897a75f6fb1" is_pre_election: true
I20260809 09:27:59.053301 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.049005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:35035 (local address 127.9.102.3:41209)
0809 09:27:59.049147 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:59.049151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:59.049163 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:27:59.049202 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:59.049205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:59.049255 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:59.049319 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:27:59.049777 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.050389 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.051644 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.052296 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.052378 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.052441 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:59.053094 (+ 653us) server_negotiation.cc:299] Negotiation successful
0809 09:27:59.053165 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:27:59.053407 21502 raft_consensus.cc:2454] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84f801fb831045dab3d877042086156a in term 0.
I20260809 09:27:59.053840 21976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767467d34aed4160a1d075eb8a272722" candidate_uuid: "84f801fb831045dab3d877042086156a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1228037bffae4bb1be226e7a9c103953" is_pre_election: true
I20260809 09:27:59.054085 21976 raft_consensus.cc:2454] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84f801fb831045dab3d877042086156a in term 0.
I20260809 09:27:59.054399 21627 leader_election.cc:302] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060f8bc83055420a98a44897a75f6fb1, 84f801fb831045dab3d877042086156a; no voters:
I20260809 09:27:59.054733 22124 raft_consensus.cc:2781] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:27:59.054847 22124 raft_consensus.cc:482] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:27:59.054906 22124 raft_consensus.cc:3037] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:59.056062 22124 raft_consensus.cc:504] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } }
I20260809 09:27:59.056429 22124 leader_election.cc:288] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [CANDIDATE]: Term 1 election: Requested vote from peers 060f8bc83055420a98a44897a75f6fb1 (127.9.102.1:38807), 1228037bffae4bb1be226e7a9c103953 (127.9.102.3:41209)
I20260809 09:27:59.056746 21976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767467d34aed4160a1d075eb8a272722" candidate_uuid: "84f801fb831045dab3d877042086156a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1228037bffae4bb1be226e7a9c103953"
I20260809 09:27:59.056847 21502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767467d34aed4160a1d075eb8a272722" candidate_uuid: "84f801fb831045dab3d877042086156a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060f8bc83055420a98a44897a75f6fb1"
I20260809 09:27:59.056857 21976 raft_consensus.cc:3037] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:59.056959 21502 raft_consensus.cc:3037] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:27:59.057914 21976 raft_consensus.cc:2454] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84f801fb831045dab3d877042086156a in term 1.
I20260809 09:27:59.058063 21502 raft_consensus.cc:2454] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84f801fb831045dab3d877042086156a in term 1.
I20260809 09:27:59.058161 21629 leader_election.cc:302] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1228037bffae4bb1be226e7a9c103953, 84f801fb831045dab3d877042086156a; no voters:
I20260809 09:27:59.058318 22124 raft_consensus.cc:2781] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:27:59.058420 22124 raft_consensus.cc:686] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 1 LEADER]: Becoming Leader. State: Replica: 84f801fb831045dab3d877042086156a, State: Running, Role: LEADER
I20260809 09:27:59.058683 22124 consensus_queue.cc:237] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } }
I20260809 09:27:59.061266 20011 catalog_manager.cc:5495] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a reported cstate change: term changed from 0 to 1, leader changed from <none> to 84f801fb831045dab3d877042086156a (127.9.102.2). New cstate: current_term: 1 leader_uuid: "84f801fb831045dab3d877042086156a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } health_report { overall_health: UNKNOWN } } }
I20260809 09:27:59.074792 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.072038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.9.102.2:34507)
0809 09:27:59.072151 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:59.072154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:59.072171 (+ 17us) server_negotiation.cc:407] Connection header received
0809 09:27:59.072267 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:59.072269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:59.072307 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:59.072403 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:27:59.072652 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.073060 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.073763 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.073937 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.074002 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.074228 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:27:59.074536 (+ 308us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:27:59.074611 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:59.074661 (+ 50us) server_negotiation.cc:299] Negotiation successful
0809 09:27:59.074699 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 09:27:59.079759 21502 raft_consensus.cc:1261] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1 [term 1 FOLLOWER]: Refusing update from remote peer 84f801fb831045dab3d877042086156a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:59.079759 21976 raft_consensus.cc:1261] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953 [term 1 FOLLOWER]: Refusing update from remote peer 84f801fb831045dab3d877042086156a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:27:59.080502 22124 consensus_queue.cc:1035] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:59.080664 22126 consensus_queue.cc:1035] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [LEADER]: Connected to new peer: Peer: permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:27:59.086748 22136 mvcc.cc:205] Tried to move back new op lower bound from 7316552413504774144 to 7316552413426778112. Current Snapshot: MvccSnapshot[applied={T|T < 7316552413504774144}]
W20260809 09:27:59.087144 21586 tablet.cc:2369] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:27:59.100628 20011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:59.100765 20011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:27:59.106386 20011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1ef11a47c3845b09d02d5a1472c860e: Sending DeleteTablet for 3 replicas of tablet 767467d34aed4160a1d075eb8a272722
I20260809 09:27:59.107298 21707 tablet_service.cc:1514] Processing DeleteTablet for tablet 767467d34aed4160a1d075eb8a272722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:59 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:41196
I20260809 09:27:59.107791 21943 tablet_service.cc:1514] Processing DeleteTablet for tablet 767467d34aed4160a1d075eb8a272722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:59 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:44074
I20260809 09:27:59.108228 21470 tablet_service.cc:1514] Processing DeleteTablet for tablet 767467d34aed4160a1d075eb8a272722 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:27:59 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:44814
I20260809 09:27:59.108552 22151 tablet_replica.cc:330] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953: stopping tablet replica
I20260809 09:27:59.108893 22151 raft_consensus.cc:2229] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:59.109128 22151 raft_consensus.cc:2258] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:59.110396 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 19957
I20260809 09:27:59.112087 22155 tablet_replica.cc:330] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1: stopping tablet replica
I20260809 09:27:59.112434 22155 raft_consensus.cc:2229] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:27:59.112742 22155 raft_consensus.cc:2258] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:59.113233 22149 tablet_replica.cc:330] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a: stopping tablet replica
I20260809 09:27:59.113554 22149 raft_consensus.cc:2229] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:27:59.113935 22149 raft_consensus.cc:2258] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:27:59.118949 22155 ts_tablet_manager.cc:1905] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:59.122982 22086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.108977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.9.102.61:46537)
0809 09:27:59.109174 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:59.109178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:59.111811 (+ 2633us) server_negotiation.cc:407] Connection header received
0809 09:27:59.111933 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:59.111936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:59.111988 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:59.112087 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:59.113443 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.114685 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.116594 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.117009 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.117096 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.119211 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:59.119215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:59.119218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:59.119242 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:59.121031 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:59.121565 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:59.121568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:59.121570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:59.121672 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:59.121977 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:59.121980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:59.121982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:59.122177 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:59.122322 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:59.122679 (+ 357us) server_negotiation.cc:299] Negotiation successful
0809 09:27:59.122831 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260809 09:27:59.123009 22155 ts_tablet_manager.cc:1918] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:27:59.123131 22155 log.cc:1196] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-0/wal/wals/767467d34aed4160a1d075eb8a272722
I20260809 09:27:59.123617 22155 ts_tablet_manager.cc:1939] T 767467d34aed4160a1d075eb8a272722 P 060f8bc83055420a98a44897a75f6fb1: Deleting consensus metadata
I20260809 09:27:59.126299 22149 ts_tablet_manager.cc:1905] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:59.129647 22151 ts_tablet_manager.cc:1905] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:27:59.130010 22149 ts_tablet_manager.cc:1918] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:27:59.130098 22149 log.cc:1196] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/767467d34aed4160a1d075eb8a272722
I20260809 09:27:59.130467 22149 ts_tablet_manager.cc:1939] T 767467d34aed4160a1d075eb8a272722 P 84f801fb831045dab3d877042086156a: Deleting consensus metadata
I20260809 09:27:59.132704 22151 ts_tablet_manager.cc:1918] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:27:59.132826 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.109063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.9.102.60:40201)
0809 09:27:59.109256 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:59.109259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:59.123572 (+ 14313us) server_negotiation.cc:407] Connection header received
0809 09:27:59.123738 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:59.123741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:59.123784 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:59.123828 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:59.124722 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.125588 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.126784 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.127133 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.127210 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.128907 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:59.128911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:59.128914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:59.128940 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:59.130758 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:59.131284 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:59.131288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:59.131289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:59.131385 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:59.131676 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:59.131680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:59.131682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:59.131880 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:59.131991 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:59.132536 (+ 545us) server_negotiation.cc:299] Negotiation successful
0809 09:27:59.132657 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260809 09:27:59.132826 22151 log.cc:1196] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/767467d34aed4160a1d075eb8a272722
I20260809 09:27:59.134258 22151 ts_tablet_manager.cc:1939] T 767467d34aed4160a1d075eb8a272722 P 1228037bffae4bb1be226e7a9c103953: Deleting consensus metadata
W20260809 09:27:59.139183 21868 connection.cc:535] server connection from 127.0.0.1:44074 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 09:27:59.139235 21403 connection.cc:535] server connection from 127.0.0.1:44814 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 09:27:59.139230 21629 connection.cc:535] server connection from 127.0.0.1:41196 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 09:27:59.151568 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.142925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45196 (local address 127.9.102.60:40201)
0809 09:27:59.143050 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:59.143054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:59.143385 (+ 331us) server_negotiation.cc:407] Connection header received
0809 09:27:59.143580 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:59.143584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:59.143623 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:59.143689 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:59.144617 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.145351 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.146415 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.146716 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.146811 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.148270 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:59.148274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:59.148276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:59.148299 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:59.149878 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:59.150413 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:59.150416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:59.150418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:59.150507 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:59.150802 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:59.150805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:59.150806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:59.150970 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:59.151095 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:59.151310 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 09:27:59.151405 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:27:59.151678 22086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:27:59.142533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.9.102.61:46537)
0809 09:27:59.142644 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 09:27:59.142648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:27:59.143328 (+ 680us) server_negotiation.cc:407] Connection header received
0809 09:27:59.143522 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:27:59.143528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:27:59.143579 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:27:59.143653 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:27:59.144684 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.145170 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.146262 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:27:59.146531 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:27:59.146582 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:27:59.148285 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:27:59.148289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:27:59.148291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:27:59.148313 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:27:59.149968 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:59.150455 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:59.150461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:59.150465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:59.150559 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:27:59.150839 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:27:59.150842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:27:59.150844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:27:59.151030 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:27:59.151118 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 09:27:59.151440 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 09:27:59.151546 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 09:28:00.093073 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.092564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.2:55249)
0809 09:28:00.092879 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.092973 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":72,"threads_started":1}
W20260809 09:28:00.093436 21826 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:37729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
I20260809 09:28:00.094164 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.093819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.1:37437)
0809 09:28:00.094008 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.094081 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":56,"threads_started":1}
W20260809 09:28:00.094435 21581 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:37729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
I20260809 09:28:00.097282 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.096738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.3:36069)
0809 09:28:00.097078 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.097164 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":82,"threads_started":1}
W20260809 09:28:00.097537 22058 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:37729 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
I20260809 09:28:00.606967 22182 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1ef11a47c3845b09d02d5a1472c860e)
I20260809 09:28:00.607070 22183 raft_consensus.cc:482] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c1ef11a47c3845b09d02d5a1472c860e)
I20260809 09:28:00.607214 22182 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } }
I20260809 09:28:00.607255 22183 raft_consensus.cc:504] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } }
I20260809 09:28:00.608388 22183 leader_election.cc:288] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729), fb31c6aa350c4812a44f6ad842d571cd (127.9.102.60:40201)
I20260809 09:28:00.608413 22182 leader_election.cc:288] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729), edc35643011b4a2e8ee9a33dac0243d2 (127.9.102.61:46537)
I20260809 09:28:00.608642 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.608205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:36678)
0809 09:28:00.608477 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.608547 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":59,"threads_started":1}
I20260809 09:28:00.608925 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.608220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:36688)
0809 09:28:00.608775 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.608841 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":67,"threads_started":1}
W20260809 09:28:00.608896 20945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
W20260809 09:28:00.609115 21263 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
I20260809 09:28:00.609843 22184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.609640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:36696)
0809 09:28:00.609751 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.609790 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 09:28:00.609910 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.609680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:36706)
0809 09:28:00.609816 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.609854 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260809 09:28:00.609939 20945 leader_election.cc:334] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729): Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
W20260809 09:28:00.609998 21263 leader_election.cc:334] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729): Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
I20260809 09:28:00.612131 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.608245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:40201 (local address 127.0.0.1:58022)
0809 09:28:00.608552 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.608563 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:00.608649 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:00.609091 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:00.609094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:00.609111 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:00.609225 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:00.609232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:00.610326 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:00.610330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:00.611038 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:00.611046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:00.611636 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:00.611639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:00.611764 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:00.611782 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 09:28:00.611863 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 09:28:00.611938 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":21,"spinlock_wait_cycles":8832,"thread_start_us":49,"threads_started":1}
I20260809 09:28:00.612136 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.608267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:46537 (local address 127.0.0.1:46952)
0809 09:28:00.608819 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.608832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:00.608902 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:00.609279 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:00.609282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:00.609298 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:00.609434 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:00.609441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:00.610353 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:00.610356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:00.611046 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:00.611055 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:00.611661 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:00.611664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:00.611751 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:00.611772 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 09:28:00.611866 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 09:28:00.611957 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":9,"thread_start_us":51,"threads_started":1}
I20260809 09:28:00.612135 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.608575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.9.102.61:46537)
0809 09:28:00.609062 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:00.609066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:00.609077 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:28:00.609118 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:00.609123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:00.609170 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:00.609547 (+ 377us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:00.609639 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:00.610252 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:00.611166 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:00.611541 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:00.611665 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:00.611767 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:00.611880 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 09:28:00.611955 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":432,"thread_start_us":46,"threads_started":1}
I20260809 09:28:00.612142 22188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.608568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.9.102.60:40201)
0809 09:28:00.608781 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:00.608786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:00.608800 (+ 14us) server_negotiation.cc:407] Connection header received
0809 09:28:00.608843 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:00.608845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:00.608885 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:00.608964 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:00.609348 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:00.610224 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:00.611154 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:00.611521 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:00.611625 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:00.611689 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:00.611881 (+ 192us) server_negotiation.cc:299] Negotiation successful
0809 09:28:00.611978 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":150,"thread_start_us":47,"threads_started":1}
I20260809 09:28:00.612635 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb31c6aa350c4812a44f6ad842d571cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "edc35643011b4a2e8ee9a33dac0243d2" is_pre_election: true
I20260809 09:28:00.612660 21309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edc35643011b4a2e8ee9a33dac0243d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb31c6aa350c4812a44f6ad842d571cd" is_pre_election: true
I20260809 09:28:00.612803 20983 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb31c6aa350c4812a44f6ad842d571cd in term 1.
I20260809 09:28:00.612829 21309 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edc35643011b4a2e8ee9a33dac0243d2 in term 1.
I20260809 09:28:00.613106 21257 leader_election.cc:302] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: edc35643011b4a2e8ee9a33dac0243d2, fb31c6aa350c4812a44f6ad842d571cd; no voters: c1ef11a47c3845b09d02d5a1472c860e
I20260809 09:28:00.613150 20941 leader_election.cc:302] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: edc35643011b4a2e8ee9a33dac0243d2, fb31c6aa350c4812a44f6ad842d571cd; no voters: c1ef11a47c3845b09d02d5a1472c860e
I20260809 09:28:00.613373 22183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 09:28:00.613440 22183 raft_consensus.cc:482] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1ef11a47c3845b09d02d5a1472c860e)
I20260809 09:28:00.613449 22182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 09:28:00.613493 22183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 1 FOLLOWER]: Advancing to term 2
I20260809 09:28:00.613495 22182 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 FOLLOWER]: Starting leader election (detected failure of leader c1ef11a47c3845b09d02d5a1472c860e)
I20260809 09:28:00.613548 22182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 1 FOLLOWER]: Advancing to term 2
I20260809 09:28:00.614648 22182 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } }
I20260809 09:28:00.614648 22183 raft_consensus.cc:504] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } }
I20260809 09:28:00.614976 22182 leader_election.cc:288] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [CANDIDATE]: Term 2 election: Requested vote from peers c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729), edc35643011b4a2e8ee9a33dac0243d2 (127.9.102.61:46537)
I20260809 09:28:00.614980 22183 leader_election.cc:288] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [CANDIDATE]: Term 2 election: Requested vote from peers c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729), fb31c6aa350c4812a44f6ad842d571cd (127.9.102.60:40201)
I20260809 09:28:00.615283 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.615079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:36720)
0809 09:28:00.615186 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.615231 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 09:28:00.615393 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb31c6aa350c4812a44f6ad842d571cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "edc35643011b4a2e8ee9a33dac0243d2"
I20260809 09:28:00.615475 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.615179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:36722)
0809 09:28:00.615373 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.615418 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260809 09:28:00.615553 20983 raft_consensus.cc:2379] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fb31c6aa350c4812a44f6ad842d571cd in current term 2: Already voted for candidate edc35643011b4a2e8ee9a33dac0243d2 in this term.
I20260809 09:28:00.615281 21309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edc35643011b4a2e8ee9a33dac0243d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fb31c6aa350c4812a44f6ad842d571cd"
I20260809 09:28:00.615801 21309 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate edc35643011b4a2e8ee9a33dac0243d2 in current term 2: Already voted for candidate fb31c6aa350c4812a44f6ad842d571cd in this term.
I20260809 09:28:00.616034 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.615863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:36736)
0809 09:28:00.615929 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.615984 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260809 09:28:00.616142 20945 leader_election.cc:334] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729): Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
I20260809 09:28:00.616415 20941 leader_election.cc:302] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edc35643011b4a2e8ee9a33dac0243d2; no voters: c1ef11a47c3845b09d02d5a1472c860e, fb31c6aa350c4812a44f6ad842d571cd
I20260809 09:28:00.616575 22183 raft_consensus.cc:2726] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 09:28:00.616797 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:00.616572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:36748)
0809 09:28:00.616706 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:00.616748 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260809 09:28:00.616894 21263 leader_election.cc:334] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729): Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
I20260809 09:28:00.616951 21263 leader_election.cc:302] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb31c6aa350c4812a44f6ad842d571cd; no voters: c1ef11a47c3845b09d02d5a1472c860e, edc35643011b4a2e8ee9a33dac0243d2
I20260809 09:28:00.617064 22182 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 09:28:01.094455 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:01.093997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.2:58987)
0809 09:28:01.094309 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:01.094375 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":81,"threads_started":1}
I20260809 09:28:01.095283 22195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:01.094930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.1:59171)
0809 09:28:01.095155 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:01.095207 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260809 09:28:01.098593 22197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:01.098171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.3:56893)
0809 09:28:01.098416 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:01.098478 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
I20260809 09:28:02.095587 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.095187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.2:49051)
0809 09:28:02.095448 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.095501 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260809 09:28:02.096274 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.095871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.1:51517)
0809 09:28:02.096111 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.096192 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260809 09:28:02.099696 22203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.099198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.3:43875)
0809 09:28:02.099501 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.099566 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":72,"threads_started":1}
I20260809 09:28:02.517076 22204 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:28:02.517215 22204 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } }
I20260809 09:28:02.517573 22204 leader_election.cc:288] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729), edc35643011b4a2e8ee9a33dac0243d2 (127.9.102.61:46537)
I20260809 09:28:02.517899 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb31c6aa350c4812a44f6ad842d571cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "edc35643011b4a2e8ee9a33dac0243d2" is_pre_election: true
I20260809 09:28:02.518041 20983 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb31c6aa350c4812a44f6ad842d571cd in term 2.
I20260809 09:28:02.518138 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.517693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:37022)
0809 09:28:02.517935 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.518017 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
I20260809 09:28:02.518368 21257 leader_election.cc:302] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: edc35643011b4a2e8ee9a33dac0243d2, fb31c6aa350c4812a44f6ad842d571cd; no voters:
I20260809 09:28:02.518546 22204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 09:28:02.518620 22204 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:28:02.518690 22204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 2 FOLLOWER]: Advancing to term 3
I20260809 09:28:02.518921 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.518718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:37036)
0809 09:28:02.518821 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.518869 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 09:28:02.519025 21263 leader_election.cc:334] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729): Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
I20260809 09:28:02.519773 22204 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } }
I20260809 09:28:02.520064 22204 leader_election.cc:288] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [CANDIDATE]: Term 3 election: Requested vote from peers c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729), edc35643011b4a2e8ee9a33dac0243d2 (127.9.102.61:46537)
I20260809 09:28:02.520494 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.520235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:37048)
0809 09:28:02.520395 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.520442 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 09:28:02.520478 20983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb31c6aa350c4812a44f6ad842d571cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "edc35643011b4a2e8ee9a33dac0243d2"
I20260809 09:28:02.520592 20983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 2 FOLLOWER]: Advancing to term 3
I20260809 09:28:02.521131 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.520935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:37054)
0809 09:28:02.521041 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.521085 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 09:28:02.521234 21263 leader_election.cc:334] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729): Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
I20260809 09:28:02.521589 20983 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb31c6aa350c4812a44f6ad842d571cd in term 3.
I20260809 09:28:02.521870 21257 leader_election.cc:302] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: edc35643011b4a2e8ee9a33dac0243d2, fb31c6aa350c4812a44f6ad842d571cd; no voters: c1ef11a47c3845b09d02d5a1472c860e
I20260809 09:28:02.522037 22204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 3 FOLLOWER]: Leader election won for term 3
I20260809 09:28:02.522223 22204 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [term 3 LEADER]: Becoming Leader. State: Replica: fb31c6aa350c4812a44f6ad842d571cd, State: Running, Role: LEADER
I20260809 09:28:02.522500 22204 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } }
I20260809 09:28:02.523840 22210 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb31c6aa350c4812a44f6ad842d571cd. Latest consensus state: current_term: 3 leader_uuid: "fb31c6aa350c4812a44f6ad842d571cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } } }
I20260809 09:28:02.523947 22210 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:02.524246 22212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:28:02.526413 22212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:28:02.526718 22212 catalog_manager.cc:1259] Loaded cluster ID: 893b9bc4eb0343f08d68c4feac02e2a7
I20260809 09:28:02.526758 22212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:28:02.527225 22212 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:28:02.527583 22212 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: Loaded TSK: 0
I20260809 09:28:02.528287 22212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:28:02.579999 21287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:28:02.582760 21287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:28:02.585597 20983 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [term 3 FOLLOWER]: Refusing update from remote peer fb31c6aa350c4812a44f6ad842d571cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260809 09:28:02.585633 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.585380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:37084)
0809 09:28:02.585524 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.585570 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 09:28:02.586053 22210 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 09:28:02.586319 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.586082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.0.0.1:37088)
0809 09:28:02.586196 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.586256 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 09:28:02.586550 21263 consensus_peers.cc:487] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd -> Peer c1ef11a47c3845b09d02d5a1472c860e (127.9.102.62:37729): Couldn't send request to peer c1ef11a47c3845b09d02d5a1472c860e. Status: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 09:28:02.587517 22215 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb31c6aa350c4812a44f6ad842d571cd. Latest consensus state: current_term: 3 leader_uuid: "fb31c6aa350c4812a44f6ad842d571cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } } }
I20260809 09:28:02.587669 22215 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:28:02.588725 22204 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fb31c6aa350c4812a44f6ad842d571cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } } }
I20260809 09:28:02.588831 22204 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:02.588958 22216 mvcc.cc:205] Tried to move back new op lower bound from 7316552427866087424 to 7316552427614449664. Current Snapshot: MvccSnapshot[applied={T|T < 7316552427866087424}]
I20260809 09:28:02.589098 22215 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fb31c6aa350c4812a44f6ad842d571cd" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c1ef11a47c3845b09d02d5a1472c860e" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37729 } } peers { permanent_uuid: "edc35643011b4a2e8ee9a33dac0243d2" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 46537 } attrs { promote: false } } peers { permanent_uuid: "fb31c6aa350c4812a44f6ad842d571cd" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 40201 } attrs { promote: false } } }
I20260809 09:28:02.589210 22215 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc35643011b4a2e8ee9a33dac0243d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:28:02.599058 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.594868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34507 (local address 127.0.0.1:51894)
0809 09:28:02.595169 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.595181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:02.595258 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:02.595699 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:02.595701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:02.595717 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:02.595795 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:02.595802 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.596908 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.596911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:02.598013 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:02.598043 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.598704 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.598707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:02.598780 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:02.598800 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 09:28:02.598892 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 09:28:02.598952 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":72,"thread_start_us":70,"threads_started":1}
I20260809 09:28:02.599092 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.595245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.9.102.2:34507)
0809 09:28:02.595446 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:02.595476 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:02.595490 (+ 14us) server_negotiation.cc:407] Connection header received
0809 09:28:02.595542 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:02.595544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:02.595586 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:02.595893 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:02.596021 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.596763 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.598153 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.598594 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.598695 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:02.598753 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:02.598907 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 09:28:02.598974 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260809 09:28:02.599184 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.594340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38807 (local address 127.0.0.1:46780)
0809 09:28:02.594476 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.594490 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:02.594576 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:02.595509 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:02.595514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:02.595529 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:02.595623 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:02.595631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.596659 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.596662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:02.597973 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:02.597983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.598868 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.598871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:02.598955 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:02.598973 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 09:28:02.599027 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 09:28:02.599083 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":47}
I20260809 09:28:02.599601 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.594451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.9.102.1:38807)
0809 09:28:02.595017 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:02.595022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:02.595035 (+ 13us) server_negotiation.cc:407] Connection header received
0809 09:28:02.595339 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:02.595342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:02.595399 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:02.595490 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:02.595741 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.596481 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.598117 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.598681 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.598765 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:02.598817 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:02.599393 (+ 576us) server_negotiation.cc:299] Negotiation successful
0809 09:28:02.599466 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":72,"threads_started":1}
I20260809 09:28:02.599812 21707 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac0ae88b0954bb4a805c10bc67b5833 (DEFAULT_TABLE table=test-table [id=e78eaec10c6b4f6e9a93b6be8b8e0fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:02.600080 21707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac0ae88b0954bb4a805c10bc67b5833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:02.600064 21470 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac0ae88b0954bb4a805c10bc67b5833 (DEFAULT_TABLE table=test-table [id=e78eaec10c6b4f6e9a93b6be8b8e0fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:02.600294 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.594581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:41209 (local address 127.0.0.1:51420)
0809 09:28:02.594803 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.594817 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:02.594912 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:02.596030 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:02.596033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:02.596050 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:02.596164 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:02.596170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.597280 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.597284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:02.598844 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:02.598863 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.599797 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.599800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:02.600002 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:02.600020 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 09:28:02.600124 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 09:28:02.600190 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":77,"threads_started":1}
I20260809 09:28:02.600306 21470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac0ae88b0954bb4a805c10bc67b5833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:02.600350 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.595183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.9.102.3:41209)
0809 09:28:02.595712 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:02.595717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:02.595732 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:28:02.595782 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:02.595787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:02.595829 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:02.595920 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:02.596307 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.597138 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.598970 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.599534 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.599664 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:02.599736 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:02.600162 (+ 426us) server_negotiation.cc:299] Negotiation successful
0809 09:28:02.600239 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":67,"threads_started":1}
I20260809 09:28:02.602631 22233 tablet_bootstrap.cc:492] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a: Bootstrap starting.
I20260809 09:28:02.602641 22232 tablet_bootstrap.cc:492] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1: Bootstrap starting.
I20260809 09:28:02.603029 21943 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac0ae88b0954bb4a805c10bc67b5833 (DEFAULT_TABLE table=test-table [id=e78eaec10c6b4f6e9a93b6be8b8e0fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:02.603258 21943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac0ae88b0954bb4a805c10bc67b5833. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:02.603492 22232 tablet_bootstrap.cc:654] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:02.603621 22233 tablet_bootstrap.cc:654] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:02.604750 22233 tablet_bootstrap.cc:492] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a: No bootstrap required, opened a new log
I20260809 09:28:02.604818 22233 ts_tablet_manager.cc:1397] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:28:02.605196 22233 raft_consensus.cc:348] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } }
I20260809 09:28:02.605274 22233 raft_consensus.cc:374] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:02.605293 22233 raft_consensus.cc:729] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f801fb831045dab3d877042086156a, State: Initialized, Role: FOLLOWER
I20260809 09:28:02.605413 22233 consensus_queue.cc:260] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } }
I20260809 09:28:02.605710 22243 tablet_bootstrap.cc:492] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953: Bootstrap starting.
I20260809 09:28:02.605746 22233 ts_tablet_manager.cc:1428] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:28:02.606292 22232 tablet_bootstrap.cc:492] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1: No bootstrap required, opened a new log
I20260809 09:28:02.606374 22232 ts_tablet_manager.cc:1397] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 09:28:02.606783 22232 raft_consensus.cc:348] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } }
I20260809 09:28:02.606894 22232 raft_consensus.cc:374] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:02.606925 22232 raft_consensus.cc:729] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060f8bc83055420a98a44897a75f6fb1, State: Initialized, Role: FOLLOWER
I20260809 09:28:02.607055 22232 consensus_queue.cc:260] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } }
I20260809 09:28:02.607344 22232 ts_tablet_manager.cc:1428] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:28:02.607371 22243 tablet_bootstrap.cc:654] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:02.607507 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.606343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.2:44661)
0809 09:28:02.607382 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.607434 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":971,"thread_start_us":79,"threads_started":1}
I20260809 09:28:02.607869 21824 heartbeater.cc:503] Master 127.9.102.60:40201 was elected leader, sending a full tablet report...
I20260809 09:28:02.608474 22243 tablet_bootstrap.cc:492] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953: No bootstrap required, opened a new log
I20260809 09:28:02.608557 22243 ts_tablet_manager.cc:1397] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260809 09:28:02.608887 22243 raft_consensus.cc:348] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } }
I20260809 09:28:02.608987 22243 raft_consensus.cc:374] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:02.609022 22243 raft_consensus.cc:729] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1228037bffae4bb1be226e7a9c103953, State: Initialized, Role: FOLLOWER
I20260809 09:28:02.609148 22243 consensus_queue.cc:260] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } }
I20260809 09:28:02.609402 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.608045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.1:36839)
0809 09:28:02.609265 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.609318 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":103,"threads_started":1}
I20260809 09:28:02.609449 22243 ts_tablet_manager.cc:1428] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:28:02.610090 21580 heartbeater.cc:503] Master 127.9.102.60:40201 was elected leader, sending a full tablet report...
I20260809 09:28:02.610633 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.609875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.1:37825)
0809 09:28:02.610553 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.610589 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":627}
I20260809 09:28:02.610633 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.609286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.2:60461)
0809 09:28:02.610534 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.610582 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1187}
I20260809 09:28:02.611107 22054 heartbeater.cc:503] Master 127.9.102.60:40201 was elected leader, sending a full tablet report...
I20260809 09:28:02.611225 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.609979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.3:57439)
0809 09:28:02.610952 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.611000 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":918,"thread_start_us":81,"threads_started":1}
I20260809 09:28:02.612445 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.612231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.3:40095)
0809 09:28:02.612338 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.612392 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 09:28:02.723562 22061 tablet.cc:2369] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:28:02.762950 21832 tablet.cc:2369] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:28:02.830289 22257 raft_consensus.cc:482] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:28:02.830427 22257 raft_consensus.cc:504] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } }
I20260809 09:28:02.831524 22257 leader_election.cc:288] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 060f8bc83055420a98a44897a75f6fb1 (127.9.102.1:38807), 84f801fb831045dab3d877042086156a (127.9.102.2:34507)
I20260809 09:28:02.835062 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.831779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:33847 (local address 127.9.102.1:38807)
0809 09:28:02.831885 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:02.831892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:02.831976 (+ 84us) server_negotiation.cc:407] Connection header received
0809 09:28:02.832165 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:02.832171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:02.832212 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:02.832289 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:02.832536 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.833020 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.834148 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.834514 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.834620 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:02.834675 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:02.834845 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 09:28:02.834912 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 09:28:02.835062 22258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.831706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38807 (local address 127.9.102.3:33847)
0809 09:28:02.831883 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.831896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:02.832023 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:02.832300 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:02.832303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:02.832319 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:02.832432 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:02.832437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.833139 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.833143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:02.834022 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:02.834031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.834626 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.834629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:02.834712 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:02.834739 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 09:28:02.834820 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 09:28:02.834884 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":42}
I20260809 09:28:02.835091 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.831857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:41645 (local address 127.9.102.2:34507)
0809 09:28:02.831981 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:02.831985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:02.832230 (+ 245us) server_negotiation.cc:407] Connection header received
0809 09:28:02.832367 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:02.832370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:02.832408 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:02.832482 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:02.832743 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.833183 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.834212 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.834572 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.834675 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:02.834726 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:02.834881 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 09:28:02.834964 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 09:28:02.835124 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.831735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34507 (local address 127.9.102.3:41645)
0809 09:28:02.832134 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.832146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:02.832256 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:02.832493 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:02.832496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:02.832509 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:02.832607 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:02.832617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.833310 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.833315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:02.834093 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:02.834101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.834680 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.834683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:02.834758 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:02.834776 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 09:28:02.834867 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 09:28:02.834931 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":19,"thread_start_us":119,"threads_started":1}
I20260809 09:28:02.835474 21502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac0ae88b0954bb4a805c10bc67b5833" candidate_uuid: "1228037bffae4bb1be226e7a9c103953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060f8bc83055420a98a44897a75f6fb1" is_pre_election: true
I20260809 09:28:02.835564 21739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac0ae88b0954bb4a805c10bc67b5833" candidate_uuid: "1228037bffae4bb1be226e7a9c103953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f801fb831045dab3d877042086156a" is_pre_election: true
I20260809 09:28:02.835629 21502 raft_consensus.cc:2454] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1228037bffae4bb1be226e7a9c103953 in term 0.
I20260809 09:28:02.835783 21739 raft_consensus.cc:2454] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1228037bffae4bb1be226e7a9c103953 in term 0.
I20260809 09:28:02.836035 21865 leader_election.cc:302] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060f8bc83055420a98a44897a75f6fb1, 1228037bffae4bb1be226e7a9c103953; no voters:
I20260809 09:28:02.836370 22257 raft_consensus.cc:2781] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:28:02.836447 22257 raft_consensus.cc:482] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:28:02.836498 22257 raft_consensus.cc:3037] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:02.837539 22257 raft_consensus.cc:504] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } }
I20260809 09:28:02.837870 22257 leader_election.cc:288] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [CANDIDATE]: Term 1 election: Requested vote from peers 060f8bc83055420a98a44897a75f6fb1 (127.9.102.1:38807), 84f801fb831045dab3d877042086156a (127.9.102.2:34507)
I20260809 09:28:02.838146 21502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac0ae88b0954bb4a805c10bc67b5833" candidate_uuid: "1228037bffae4bb1be226e7a9c103953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "060f8bc83055420a98a44897a75f6fb1"
I20260809 09:28:02.838165 21739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ac0ae88b0954bb4a805c10bc67b5833" candidate_uuid: "1228037bffae4bb1be226e7a9c103953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84f801fb831045dab3d877042086156a"
I20260809 09:28:02.838248 21739 raft_consensus.cc:3037] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:02.838268 21502 raft_consensus.cc:3037] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:02.839073 21502 raft_consensus.cc:2454] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1228037bffae4bb1be226e7a9c103953 in term 1.
I20260809 09:28:02.839118 21739 raft_consensus.cc:2454] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1228037bffae4bb1be226e7a9c103953 in term 1.
I20260809 09:28:02.839351 21865 leader_election.cc:302] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060f8bc83055420a98a44897a75f6fb1, 1228037bffae4bb1be226e7a9c103953; no voters:
I20260809 09:28:02.839545 22257 raft_consensus.cc:2781] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:28:02.839788 22257 raft_consensus.cc:686] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 1 LEADER]: Becoming Leader. State: Replica: 1228037bffae4bb1be226e7a9c103953, State: Running, Role: LEADER
I20260809 09:28:02.840175 22257 consensus_queue.cc:237] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } }
I20260809 09:28:02.840452 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.840218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.3:45583)
0809 09:28:02.840314 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.840368 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 09:28:02.842621 21287 catalog_manager.cc:5495] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1228037bffae4bb1be226e7a9c103953 (127.9.102.3). New cstate: current_term: 1 leader_uuid: "1228037bffae4bb1be226e7a9c103953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1228037bffae4bb1be226e7a9c103953" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 } health_report { overall_health: UNKNOWN } } }
I20260809 09:28:02.885658 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.882620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.9.102.3:41209)
0809 09:28:02.882746 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:02.882749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:02.882761 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:28:02.882862 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:02.882864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:02.882905 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:02.882980 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:28:02.883206 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.883667 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.884475 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:02.884658 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:02.884732 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:02.884927 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:28:02.885321 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:28:02.885410 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:02.885468 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 09:28:02.885505 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 09:28:02.888032 21502 raft_consensus.cc:1261] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1 [term 1 FOLLOWER]: Refusing update from remote peer 1228037bffae4bb1be226e7a9c103953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:02.888285 21739 raft_consensus.cc:1261] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a [term 1 FOLLOWER]: Refusing update from remote peer 1228037bffae4bb1be226e7a9c103953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:02.888549 22261 consensus_queue.cc:1035] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "060f8bc83055420a98a44897a75f6fb1" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:02.888804 22257 consensus_queue.cc:1035] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84f801fb831045dab3d877042086156a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:02.890882 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.890654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.1:54871)
0809 09:28:02.890745 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.890800 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260809 09:28:02.891607 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.891358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.2:42607)
0809 09:28:02.891455 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.891504 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260809 09:28:02.892879 22264 mvcc.cc:205] Tried to move back new op lower bound from 7316552429105102848 to 7316552428916207616. Current Snapshot: MvccSnapshot[applied={T|T < 7316552429105102848}]
I20260809 09:28:02.893479 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.893278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.3:38987)
0809 09:28:02.893360 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.893411 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260809 09:28:02.894694 22251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.894314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.1:59199)
0809 09:28:02.894584 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.894634 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260809 09:28:02.896347 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.895523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.3:60603)
0809 09:28:02.896238 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.896292 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":655}
I20260809 09:28:02.897107 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.896881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:37729 (local address 127.9.102.2:51343)
0809 09:28:02.896977 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.897046 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260809 09:28:02.904235 21287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:28:02.904376 21287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:28:02.907641 21287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb31c6aa350c4812a44f6ad842d571cd: Sending DeleteTablet for 3 replicas of tablet 7ac0ae88b0954bb4a805c10bc67b5833
I20260809 09:28:02.908290 21943 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ac0ae88b0954bb4a805c10bc67b5833 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:02 UTC) from {username='oryx', principal='oryx/127.9.102.60@KRBTEST.COM'} at 127.0.0.1:51420
I20260809 09:28:02.908489 21470 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ac0ae88b0954bb4a805c10bc67b5833 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:02 UTC) from {username='oryx', principal='oryx/127.9.102.60@KRBTEST.COM'} at 127.0.0.1:46780
I20260809 09:28:02.908761 22281 tablet_replica.cc:330] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1: stopping tablet replica
I20260809 09:28:02.909082 21707 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ac0ae88b0954bb4a805c10bc67b5833 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:02 UTC) from {username='oryx', principal='oryx/127.9.102.60@KRBTEST.COM'} at 127.0.0.1:51894
I20260809 09:28:02.909175 22280 tablet_replica.cc:330] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953: stopping tablet replica
I20260809 09:28:02.908874 22281 raft_consensus.cc:2229] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:28:02.909278 22280 raft_consensus.cc:2229] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:28:02.909368 22281 raft_consensus.cc:2258] T 7ac0ae88b0954bb4a805c10bc67b5833 P 060f8bc83055420a98a44897a75f6fb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:02.909565 22282 tablet_replica.cc:330] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a: stopping tablet replica
I20260809 09:28:02.909660 22282 raft_consensus.cc:2229] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:28:02.909657 22280 raft_consensus.cc:2258] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:02.909765 22282 raft_consensus.cc:2258] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:02.909797 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 21215
I20260809 09:28:02.910923 22280 ts_tablet_manager.cc:1905] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:02.910923 22282 ts_tablet_manager.cc:1905] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:02.913162 22280 ts_tablet_manager.cc:1918] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:28:02.913226 22280 log.cc:1196] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/7ac0ae88b0954bb4a805c10bc67b5833
I20260809 09:28:02.913492 22280 ts_tablet_manager.cc:1939] T 7ac0ae88b0954bb4a805c10bc67b5833 P 1228037bffae4bb1be226e7a9c103953: Deleting consensus metadata
I20260809 09:28:02.913496 22282 ts_tablet_manager.cc:1918] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:28:02.913552 22282 log.cc:1196] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/7ac0ae88b0954bb4a805c10bc67b5833
I20260809 09:28:02.913786 22282 ts_tablet_manager.cc:1939] T 7ac0ae88b0954bb4a805c10bc67b5833 P 84f801fb831045dab3d877042086156a: Deleting consensus metadata
I20260809 09:28:02.914115 21260 catalog_manager.cc:4841] TS 1228037bffae4bb1be226e7a9c103953 (127.9.102.3:41209): tablet 7ac0ae88b0954bb4a805c10bc67b5833 (table test-table [id=e78eaec10c6b4f6e9a93b6be8b8e0fde]) successfully deleted
I20260809 09:28:02.914283 21260 catalog_manager.cc:4841] TS 84f801fb831045dab3d877042086156a (127.9.102.2:34507): tablet 7ac0ae88b0954bb4a805c10bc67b5833 (table test-table [id=e78eaec10c6b4f6e9a93b6be8b8e0fde]) successfully deleted
I20260809 09:28:02.924430 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 21597
I20260809 09:28:02.924494 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:02.924240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38807 (local address 127.0.0.1:46784)
0809 09:28:02.924368 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:02.924419 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 09:28:02.926578 21257 catalog_manager.cc:4584] TS 060f8bc83055420a98a44897a75f6fb1 (127.9.102.1:38807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ac0ae88b0954bb4a805c10bc67b5833: Network error: Client connection negotiation failed: client connection to 127.9.102.1:38807: connect: Connection refused (error 111)
I20260809 09:28:02.933759 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 21839
I20260809 09:28:02.945873 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 20752
I20260809 09:28:02.954871 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 20925
2026-08-09T09:28:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7357 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:28:03 dist-test-slave-7vjv krb5kdc[22287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:28:03 dist-test-slave-7vjv krb5kdc[22287](info): set up 2 sockets
Aug 09 09:28:03 dist-test-slave-7vjv krb5kdc[22287](info): commencing operation
krb5kdc: starting...
W20260809 09:28:04.144335 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:28:04 dist-test-slave-7vjv krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:28:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:28:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.62.keytab.
Entry for principal oryx/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.62.keytab.
I20260809 09:28:04.258653 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:38761
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.62.keytab
--principal=oryx/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:38761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:04.363381 22307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:04.363672 22307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:04.363741 22307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:04.367527 22307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:28:04.367614 22307 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 09:28:04.367652 22307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:04.367702 22307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:04.367734 22307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:28:04.367769 22307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:28:04.372431 22307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:38761
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.62.keytab
--principal=oryx/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:38761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:04.373289 22307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:04.374255 22307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:04.382176 22319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:04.382234 22315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:04.382176 22316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:04.382355 22307 server_base.cc:1034] running on GCE node
I20260809 09:28:04.383078 22307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:04.384159 22307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:04.385360 22307 hybrid_clock.cc:648] HybridClock initialized: now 1786267684385349 us; error 27 us; skew 500 ppm
Aug 09 09:28:04 dist-test-slave-7vjv krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267684, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:04.388065 22307 init.cc:377] Logged in from keytab as oryx/127.9.102.62@KRBTEST.COM (short username oryx)
I20260809 09:28:04.388918 22307 webserver.cc:460] Webserver started at http://127.9.102.62:43071/ using document root <none> and password file <none>
I20260809 09:28:04.389385 22307 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:04.389437 22307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:04.389611 22307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:04.391472 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "bedbddb15d434b389fe0388919dabd8f"
format_stamp: "Formatted at 2026-08-09 09:28:04 on dist-test-slave-7vjv"
server_key: "0b3a6b8cbbc000c500e1b85bed77334b"
server_key_iv: "b656f25e1ecc22e8f12cd911a6da96cd"
server_key_version: "encryptionkey@0"
I20260809 09:28:04.391965 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "bedbddb15d434b389fe0388919dabd8f"
format_stamp: "Formatted at 2026-08-09 09:28:04 on dist-test-slave-7vjv"
server_key: "0b3a6b8cbbc000c500e1b85bed77334b"
server_key_iv: "b656f25e1ecc22e8f12cd911a6da96cd"
server_key_version: "encryptionkey@0"
I20260809 09:28:04.395223 22307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 09:28:04.397341 22332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:04.398378 22307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:28:04.398515 22307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "bedbddb15d434b389fe0388919dabd8f"
format_stamp: "Formatted at 2026-08-09 09:28:04 on dist-test-slave-7vjv"
server_key: "0b3a6b8cbbc000c500e1b85bed77334b"
server_key_iv: "b656f25e1ecc22e8f12cd911a6da96cd"
server_key_version: "encryptionkey@0"
I20260809 09:28:04.398619 22307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:04.416815 22307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:04.419728 22307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:04.419915 22307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:04.427258 22307 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:38761
I20260809 09:28:04.427282 22440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:38761 every 8 connection(s)
I20260809 09:28:04.428027 22307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:28:04.430305 22441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:04.434553 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 22307
I20260809 09:28:04.434713 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:28:04.435001 9624 external_mini_cluster.cc:1428] Setting key 211041a691ea2aef2acb9271c75d1961
I20260809 09:28:04.435295 22441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f: Bootstrap starting.
I20260809 09:28:04.437553 22441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:04.438230 22441 log.cc:824] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f: Log is configured to *not* fsync() on all Append() calls
I20260809 09:28:04.439903 22441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f: No bootstrap required, opened a new log
Aug 09 09:28:04 dist-test-slave-7vjv krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:28:04.443418 22441 raft_consensus.cc:348] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedbddb15d434b389fe0388919dabd8f" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38761 } }
I20260809 09:28:04.443692 22441 raft_consensus.cc:374] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:04.443753 22441 raft_consensus.cc:729] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bedbddb15d434b389fe0388919dabd8f, State: Initialized, Role: FOLLOWER
I20260809 09:28:04.444203 22441 consensus_queue.cc:260] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedbddb15d434b389fe0388919dabd8f" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38761 } }
I20260809 09:28:04.444339 22441 raft_consensus.cc:388] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:28:04.444422 22441 raft_consensus.cc:482] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:28:04.444515 22441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:04.445226 22441 raft_consensus.cc:504] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedbddb15d434b389fe0388919dabd8f" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38761 } }
I20260809 09:28:04.445549 22441 leader_election.cc:302] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bedbddb15d434b389fe0388919dabd8f; no voters:
I20260809 09:28:04.445820 22441 leader_election.cc:288] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:28:04.445935 22451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:28:04.446139 22451 raft_consensus.cc:686] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [term 1 LEADER]: Becoming Leader. State: Replica: bedbddb15d434b389fe0388919dabd8f, State: Running, Role: LEADER
I20260809 09:28:04.446434 22451 consensus_queue.cc:237] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedbddb15d434b389fe0388919dabd8f" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38761 } }
I20260809 09:28:04.446993 22441 sys_catalog.cc:564] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:28:04.447675 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bedbddb15d434b389fe0388919dabd8f. Latest consensus state: current_term: 1 leader_uuid: "bedbddb15d434b389fe0388919dabd8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedbddb15d434b389fe0388919dabd8f" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38761 } } }
I20260809 09:28:04.447793 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:04.447921 22452 sys_catalog.cc:455] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bedbddb15d434b389fe0388919dabd8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bedbddb15d434b389fe0388919dabd8f" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38761 } } }
I20260809 09:28:04.448012 22452 sys_catalog.cc:458] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:04.448390 22468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:28:04.449242 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:04.436310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.9.102.62:38761)
0809 09:28:04.436760 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:04.436773 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:04.436804 (+ 31us) server_negotiation.cc:407] Connection header received
0809 09:28:04.437350 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:04.437365 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:04.437636 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:04.437996 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:04.438756 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.439692 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.440659 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.441153 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.441331 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:04.444154 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:04.444160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:04.444174 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:04.444201 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:04.446089 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:04.446567 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:04.446573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:04.446578 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:04.446698 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:04.446995 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:04.446998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:04.447000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:04.447451 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:04.447617 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:04.448186 (+ 569us) server_negotiation.cc:299] Negotiation successful
0809 09:28:04.448446 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":145,"threads_started":1}
I20260809 09:28:04.451478 22468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:28:04.457096 22468 catalog_manager.cc:1347] Generated new cluster ID: 0833067a88644f05acfd8d974fa7cde2
I20260809 09:28:04.457198 22468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:28:04.468564 22468 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:28:04.469349 22468 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:28:04.479573 22468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f: Generated new TSK 0
I20260809 09:28:04.480300 22468 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab.
Entry for principal oryx/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab.
I20260809 09:28:04.519162 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38761
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab
--principal=oryx/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:04.622885 22489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:04.623087 22489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:04.623123 22489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:04.626699 22489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:04.626747 22489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:04.626778 22489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 09:28:04.626804 22489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:28:04.631067 22489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab
--principal=oryx/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.102.62:38761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:04.631814 22489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:04.632627 22489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:04.641438 22496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:04.641525 22497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:04.641666 22489 server_base.cc:1034] running on GCE node
W20260809 09:28:04.641803 22499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:04.642232 22489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:04.642725 22489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:04.643882 22489 hybrid_clock.cc:648] HybridClock initialized: now 1786267684643856 us; error 37 us; skew 500 ppm
Aug 09 09:28:04 dist-test-slave-7vjv krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267684, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:04.646399 22489 init.cc:377] Logged in from keytab as oryx/127.9.102.1@KRBTEST.COM (short username oryx)
I20260809 09:28:04.647106 22489 webserver.cc:460] Webserver started at http://127.9.102.1:45091/ using document root <none> and password file <none>
I20260809 09:28:04.647668 22489 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:04.647732 22489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:04.647877 22489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:04.649750 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "2cfe2644bebf4251930b0be0d02c2584"
format_stamp: "Formatted at 2026-08-09 09:28:04 on dist-test-slave-7vjv"
server_key: "efba00189705e9bfe5cfca5db8c6c3e9"
server_key_iv: "9a75e8a3230f4f76db122f04b7d330d8"
server_key_version: "encryptionkey@0"
I20260809 09:28:04.650198 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "2cfe2644bebf4251930b0be0d02c2584"
format_stamp: "Formatted at 2026-08-09 09:28:04 on dist-test-slave-7vjv"
server_key: "efba00189705e9bfe5cfca5db8c6c3e9"
server_key_iv: "9a75e8a3230f4f76db122f04b7d330d8"
server_key_version: "encryptionkey@0"
I20260809 09:28:04.653424 22489 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:28:04.655393 22512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:04.656493 22489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:28:04.656581 22489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "2cfe2644bebf4251930b0be0d02c2584"
format_stamp: "Formatted at 2026-08-09 09:28:04 on dist-test-slave-7vjv"
server_key: "efba00189705e9bfe5cfca5db8c6c3e9"
server_key_iv: "9a75e8a3230f4f76db122f04b7d330d8"
server_key_version: "encryptionkey@0"
I20260809 09:28:04.656649 22489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:04.678714 22489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:04.682006 22489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:04.682153 22489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:04.683341 22489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:04.683420 22489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:04.683488 22489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:04.683548 22489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 09:28:04 dist-test-slave-7vjv krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267684, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:28:04.693650 22489 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:38443
I20260809 09:28:04.693967 22719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:38443 every 8 connection(s)
I20260809 09:28:04.694639 22489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:28:04.694962 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 22489
I20260809 09:28:04.695091 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:28:04.695416 9624 external_mini_cluster.cc:1428] Setting key c5902a32bd2fc395cfe5e07792ece9c3
I20260809 09:28:04.696995 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:04.684177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:40773 (local address 127.9.102.62:38761)
0809 09:28:04.684304 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:04.684307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:04.685220 (+ 913us) server_negotiation.cc:407] Connection header received
0809 09:28:04.686063 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:04.686067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:04.686121 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:04.686217 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:04.687608 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.688413 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.689464 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.689690 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.689808 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:04.692996 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:04.693002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:04.693005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:04.693038 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:04.694790 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:04.695394 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:04.695397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:04.695399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:04.695485 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:04.695860 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:04.695863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:04.695864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:04.696083 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:04.696202 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:04.696740 (+ 538us) server_negotiation.cc:299] Negotiation successful
0809 09:28:04.696861 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:28:04.697889 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:04.684650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38761 (local address 127.9.102.1:40773)
0809 09:28:04.685081 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:04.685119 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:04.685855 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:04.686370 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:04.686377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:04.686849 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:04.687400 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:04.687413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.688544 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.688548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:04.689336 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:04.689345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.689853 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.689857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:04.690008 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:04.690561 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:04.690578 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:04.692818 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:04.694928 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:04.694935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:04.694951 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:04.695295 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:04.695589 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:04.695593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:04.695595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:04.695765 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:04.696258 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:04.696263 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:04.696461 (+ 198us) client_negotiation.cc:769] Sending connection context
0809 09:28:04.696661 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 09:28:04.696979 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":121,"threads_started":1}
I20260809 09:28:04.699919 22720 heartbeater.cc:347] Connected to a master server at 127.9.102.62:38761
I20260809 09:28:04.700081 22720 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:04.700637 22720 heartbeater.cc:511] Master 127.9.102.62:38761 requested a full tablet report, sending...
I20260809 09:28:04.702277 22351 ts_manager.cc:194] Registered new tserver with Master: 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443)
I20260809 09:28:04.703634 22351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:40773
WARNING: no policy specified for oryx/127.9.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.2@KRBTEST.COM" created.
I20260809 09:28:04.710343 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:04.702188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38761 (local address 127.9.102.1:45785)
0809 09:28:04.702609 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:04.702624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:04.702728 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:04.702984 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:04.702989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:04.703214 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:04.703363 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:04.703372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.704304 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.704309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:04.705262 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:04.705275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.705858 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.705862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:04.705974 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:04.706434 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:04.706448 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:04.706916 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:04.708719 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:04.708721 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:04.708722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:04.708990 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:04.709243 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:04.709245 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:04.709247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:04.709377 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:04.709878 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:04.709884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:04.709991 (+ 107us) client_negotiation.cc:769] Sending connection context
0809 09:28:04.710116 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 09:28:04.710214 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":126,"threads_started":1}
I20260809 09:28:04.710719 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:04.701992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:45785 (local address 127.9.102.62:38761)
0809 09:28:04.702105 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:04.702109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:04.702715 (+ 606us) server_negotiation.cc:407] Connection header received
0809 09:28:04.702816 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:04.702819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:04.702858 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:04.702934 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:04.703489 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.704167 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.705409 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.705722 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.705804 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:04.707043 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:04.707046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:04.707048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:04.707068 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:04.708603 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:04.709082 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:04.709085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:04.709086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:04.709157 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:04.709518 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:04.709521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:04.709531 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:04.709776 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:04.709852 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:04.710210 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 09:28:04.710337 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab.
Entry for principal oryx/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab.
I20260809 09:28:04.724373 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38761
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab
--principal=oryx/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:04.823856 22737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:04.824048 22737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:04.824085 22737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:04.827639 22737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:04.827687 22737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:04.827720 22737 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 09:28:04.827749 22737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:28:04.831996 22737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab
--principal=oryx/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.102.62:38761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:04.832825 22737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:04.833739 22737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:04.842607 22753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:04.842633 22748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:04.842640 22746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:04.842922 22737 server_base.cc:1034] running on GCE node
I20260809 09:28:04.843293 22737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:04.843842 22737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:04.845026 22737 hybrid_clock.cc:648] HybridClock initialized: now 1786267684844995 us; error 45 us; skew 500 ppm
Aug 09 09:28:04 dist-test-slave-7vjv krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267684, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:04.847812 22737 init.cc:377] Logged in from keytab as oryx/127.9.102.2@KRBTEST.COM (short username oryx)
I20260809 09:28:04.848712 22737 webserver.cc:460] Webserver started at http://127.9.102.2:34647/ using document root <none> and password file <none>
I20260809 09:28:04.849293 22737 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:04.849386 22737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:04.849582 22737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:04.851320 22737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "4869fad624c04f58927b0d002df8ce79"
format_stamp: "Formatted at 2026-08-09 09:28:04 on dist-test-slave-7vjv"
server_key: "7c91d3eee4f361eae66e422c3744e19b"
server_key_iv: "d6ad17b6662891127a373a17b3fc9a6d"
server_key_version: "encryptionkey@0"
I20260809 09:28:04.851876 22737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "4869fad624c04f58927b0d002df8ce79"
format_stamp: "Formatted at 2026-08-09 09:28:04 on dist-test-slave-7vjv"
server_key: "7c91d3eee4f361eae66e422c3744e19b"
server_key_iv: "d6ad17b6662891127a373a17b3fc9a6d"
server_key_version: "encryptionkey@0"
I20260809 09:28:04.855124 22737 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 09:28:04.857445 22764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:04.858498 22737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:28:04.858644 22737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "4869fad624c04f58927b0d002df8ce79"
format_stamp: "Formatted at 2026-08-09 09:28:04 on dist-test-slave-7vjv"
server_key: "7c91d3eee4f361eae66e422c3744e19b"
server_key_iv: "d6ad17b6662891127a373a17b3fc9a6d"
server_key_version: "encryptionkey@0"
I20260809 09:28:04.858750 22737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:04.871918 22737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:04.874786 22737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:04.874989 22737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:04.876268 22737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:04.876340 22737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:04.876400 22737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:04.876451 22737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 09:28:04 dist-test-slave-7vjv krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267684, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:28:04.886765 22737 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:35759
I20260809 09:28:04.886843 22959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:35759 every 8 connection(s)
I20260809 09:28:04.887542 22737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:28:04.889083 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:04.876993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:33035 (local address 127.9.102.62:38761)
0809 09:28:04.877113 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:04.877116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:04.877964 (+ 848us) server_negotiation.cc:407] Connection header received
0809 09:28:04.878742 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:04.878745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:04.878800 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:04.878867 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:04.880070 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.880937 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.882101 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.882458 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.882573 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:04.885602 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:04.885605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:04.885608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:04.885630 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:04.887121 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:04.887587 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:04.887591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:04.887593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:04.887673 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:04.887990 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:04.887993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:04.887994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:04.888246 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:04.888341 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:04.888855 (+ 514us) server_negotiation.cc:299] Negotiation successful
0809 09:28:04.888968 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 09:28:04.889739 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 22737
I20260809 09:28:04.889871 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:28:04.889880 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:04.877391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38761 (local address 127.9.102.2:33035)
0809 09:28:04.877831 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:04.877865 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:04.878540 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:04.879010 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:04.879017 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:04.879330 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:04.879890 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:04.879911 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.881079 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.881082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:04.881979 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:04.881988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.882604 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.882608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:04.882764 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:04.883246 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:04.883267 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:04.885442 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:04.887231 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:04.887237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:04.887248 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:04.887477 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:04.887743 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:04.887747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:04.887749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:04.887882 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:04.888389 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:04.888394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:04.888613 (+ 219us) client_negotiation.cc:769] Sending connection context
0809 09:28:04.888767 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 09:28:04.888996 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":126,"threads_started":1}
I20260809 09:28:04.890197 9624 external_mini_cluster.cc:1428] Setting key 56bbf9c4ced94bc0cc4468061d6ecbb1
I20260809 09:28:04.891142 22960 heartbeater.cc:347] Connected to a master server at 127.9.102.62:38761
I20260809 09:28:04.891260 22960 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:04.891752 22960 heartbeater.cc:511] Master 127.9.102.62:38761 requested a full tablet report, sending...
I20260809 09:28:04.892841 22351 ts_manager.cc:194] Registered new tserver with Master: 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759)
I20260809 09:28:04.893553 22351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:33035
I20260809 09:28:04.901115 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:04.893173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:33513 (local address 127.9.102.62:38761)
0809 09:28:04.893272 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:04.893276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:04.893498 (+ 222us) server_negotiation.cc:407] Connection header received
0809 09:28:04.893607 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:04.893610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:04.893646 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:04.893737 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:04.894608 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.895264 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.896471 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.896658 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.896735 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:04.897893 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:04.897895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:04.897897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:04.897916 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:04.899286 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:04.899845 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:04.899848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:04.899849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:04.899917 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:04.900295 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:04.900298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:04.900302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:04.900491 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:04.900620 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:04.900867 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 09:28:04.900966 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for oryx/127.9.102.3@KRBTEST.COM; defaulting to no policy
I20260809 09:28:04.901151 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:04.893117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38761 (local address 127.9.102.2:33513)
0809 09:28:04.893408 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:04.893418 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:04.893535 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:04.893750 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:04.893753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:04.894314 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:04.894466 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:04.894476 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.895429 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.895434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:04.896352 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:04.896366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:04.896788 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:04.896794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:04.896952 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:04.897362 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:04.897376 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:04.897762 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:04.899406 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:04.899409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:04.899413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:04.899740 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:04.900069 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:04.900072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:04.900074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:04.900187 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:04.900615 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:04.900618 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:04.900702 (+ 84us) client_negotiation.cc:769] Sending connection context
0809 09:28:04.900816 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 09:28:04.900944 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":3,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab.
Entry for principal oryx/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab.
I20260809 09:28:04.918478 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38761
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab
--principal=oryx/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:05.021785 22975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:05.022065 22975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:05.022142 22975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:05.026051 22975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:05.026129 22975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:05.026196 22975 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 09:28:05.026271 22975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:28:05.030761 22975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab
--principal=oryx/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.102.62:38761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:05.031687 22975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:05.032634 22975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:05.041770 22983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:05.041767 22984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:05.041810 22975 server_base.cc:1034] running on GCE node
W20260809 09:28:05.041766 22986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:05.042415 22975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:05.042986 22975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:05.044226 22975 hybrid_clock.cc:648] HybridClock initialized: now 1786267685044210 us; error 28 us; skew 500 ppm
Aug 09 09:28:05 dist-test-slave-7vjv krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267685, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:05.046811 22975 init.cc:377] Logged in from keytab as oryx/127.9.102.3@KRBTEST.COM (short username oryx)
I20260809 09:28:05.047662 22975 webserver.cc:460] Webserver started at http://127.9.102.3:45679/ using document root <none> and password file <none>
I20260809 09:28:05.048280 22975 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:05.048362 22975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:05.048556 22975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:05.050376 22975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "e90437df78a34b5a854338321873a93f"
format_stamp: "Formatted at 2026-08-09 09:28:05 on dist-test-slave-7vjv"
server_key: "3d6c7c3693d741c1a5d8a1a0f6fca2db"
server_key_iv: "d39d05c628747bece18b6d094988828e"
server_key_version: "encryptionkey@0"
I20260809 09:28:05.050894 22975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "e90437df78a34b5a854338321873a93f"
format_stamp: "Formatted at 2026-08-09 09:28:05 on dist-test-slave-7vjv"
server_key: "3d6c7c3693d741c1a5d8a1a0f6fca2db"
server_key_iv: "d39d05c628747bece18b6d094988828e"
server_key_version: "encryptionkey@0"
I20260809 09:28:05.054252 22975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260809 09:28:05.056358 22996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:05.057382 22975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260809 09:28:05.057544 22975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "e90437df78a34b5a854338321873a93f"
format_stamp: "Formatted at 2026-08-09 09:28:05 on dist-test-slave-7vjv"
server_key: "3d6c7c3693d741c1a5d8a1a0f6fca2db"
server_key_iv: "d39d05c628747bece18b6d094988828e"
server_key_version: "encryptionkey@0"
I20260809 09:28:05.057649 22975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:05.069062 22975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:05.072067 22975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:05.072319 22975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:05.073611 22975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:05.073690 22975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:05.073747 22975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:05.073794 22975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 09:28:05 dist-test-slave-7vjv krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267685, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:28:05.083772 22975 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:35891
I20260809 09:28:05.083860 23183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:35891 every 8 connection(s)
I20260809 09:28:05.084805 22975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:28:05.087087 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.074359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:36919 (local address 127.9.102.62:38761)
0809 09:28:05.074480 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.074483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.075321 (+ 838us) server_negotiation.cc:407] Connection header received
0809 09:28:05.076334 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.076338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.076403 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.076467 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:05.078151 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.078960 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.080785 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.081142 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.081269 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.083961 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:05.083967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:05.083973 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:05.084007 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:05.085494 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:05.085951 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:05.085954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:05.085955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:05.086030 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:05.086264 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:05.086266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:05.086267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:05.086419 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:05.086491 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.086887 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.086975 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 09:28:05.087723 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.074763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38761 (local address 127.9.102.3:36919)
0809 09:28:05.075185 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.075220 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.076052 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.076905 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.076917 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.077385 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:05.077934 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.077951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.079101 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.079104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.080629 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.080643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.081286 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.081288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.081408 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.081922 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:05.081937 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:05.083761 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:05.085618 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:05.085623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:05.085634 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:05.085862 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:05.086089 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:05.086091 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:05.086092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:05.086193 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:05.086509 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:05.086513 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:05.086699 (+ 186us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.086831 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.087023 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":124,"threads_started":1}
I20260809 09:28:05.088909 23186 heartbeater.cc:347] Connected to a master server at 127.9.102.62:38761
I20260809 09:28:05.089083 23186 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:05.089576 23186 heartbeater.cc:511] Master 127.9.102.62:38761 requested a full tablet report, sending...
I20260809 09:28:05.090610 22352 ts_manager.cc:194] Registered new tserver with Master: e90437df78a34b5a854338321873a93f (127.9.102.3:35891)
I20260809 09:28:05.091347 22352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:36919
I20260809 09:28:05.093918 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 22975
I20260809 09:28:05.094033 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:28:05.094300 9624 external_mini_cluster.cc:1428] Setting key 1746561cb9fd6beb8ff28b8adcd688f1
I20260809 09:28:05.096429 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:28:05.097553 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.090848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:46895 (local address 127.9.102.62:38761)
0809 09:28:05.090945 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.090949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.091243 (+ 294us) server_negotiation.cc:407] Connection header received
0809 09:28:05.091329 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.091330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.091365 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.091426 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:05.091943 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.092510 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.093280 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.093487 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.093597 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.094584 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:05.094587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:05.094589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:05.094617 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:05.095957 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:05.096437 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:05.096441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:05.096443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:05.096556 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:05.096813 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:05.096816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:05.096818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:05.096983 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:05.097095 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.097328 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.097420 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 09:28:05.097585 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.090880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38761 (local address 127.9.102.3:46895)
0809 09:28:05.091153 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.091166 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.091254 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.091505 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.091508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.091695 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:05.091779 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.091785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.092648 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.092651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.093159 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.093167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.093602 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.093605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.093691 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.094129 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:05.094141 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:05.094454 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:05.096080 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:05.096082 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:05.096084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:05.096335 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:05.096648 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:05.096651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:05.096652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:05.096728 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:05.097089 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:05.097092 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:05.097173 (+ 81us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.097280 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.097405 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":128,"threads_started":1}
Aug 09 09:28:05 dist-test-slave-7vjv krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267685, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 09:28:05 dist-test-slave-7vjv krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:05 dist-test-slave-7vjv krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 09:28:05.119844 22449 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:05.114424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.9.102.62:38761)
0809 09:28:05.114533 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.114537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.114549 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:28:05.114577 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.114578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.114608 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.114668 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:05.115260 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.115693 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.116556 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.116752 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.116823 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.119734 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36920: BlockingRecv error: recv got EOF from 127.0.0.1:36920 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 09:28:05 dist-test-slave-7vjv krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267685, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:28:05.128512 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.121184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.9.102.62:38761)
0809 09:28:05.121312 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.121316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.121434 (+ 118us) server_negotiation.cc:407] Connection header received
0809 09:28:05.121533 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.121538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.121578 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.121654 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:05.122176 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.122602 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.123319 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.123502 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.123603 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.125785 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:05.125788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:05.125790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:05.125810 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:05.127067 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:05.127449 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:05.127453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:05.127455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:05.127530 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:05.127790 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:05.127793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:05.127795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:05.127939 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:05.128074 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.128292 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.128386 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 09:28:05.130934 22352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:28:05.133725 22352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:28:05.146901 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.142163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.9.102.3:35891)
0809 09:28:05.142566 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.142577 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.142596 (+ 19us) server_negotiation.cc:407] Connection header received
0809 09:28:05.142665 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.142669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.142833 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.142966 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:05.143325 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.143941 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.145689 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.146236 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.146364 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.146512 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.146691 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.146780 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":105,"threads_started":1}
I20260809 09:28:05.146914 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.141232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:35759 (local address 127.0.0.1:50102)
0809 09:28:05.141760 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.141803 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.141905 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.143196 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.143199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.143210 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:05.143289 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.143296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.144427 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.144430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.145374 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.145383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.146478 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.146482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.146580 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.146600 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.146643 (+ 43us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.146780 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":55,"thread_start_us":79,"threads_started":1}
I20260809 09:28:05.146908 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.141867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:35891 (local address 127.0.0.1:36642)
0809 09:28:05.142103 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.142115 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.142188 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.143097 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.143100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.143120 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:05.143211 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.143216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.144049 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.144054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.145562 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.145570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.146367 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.146370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.146464 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.146548 (+ 84us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.146645 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.146791 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":197,"thread_start_us":85,"threads_started":1}
I20260809 09:28:05.147380 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.141451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38443 (local address 127.0.0.1:54802)
0809 09:28:05.141983 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.141997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.142090 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.142846 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.142851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.142871 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:05.142989 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.142996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.144183 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.144186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.145553 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.145565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.146569 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.146572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.146643 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.146660 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.146706 (+ 46us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.147254 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":610,"thread_start_us":89,"threads_started":1}
I20260809 09:28:05.147711 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.142058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.9.102.1:38443)
0809 09:28:05.142426 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.142435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.142452 (+ 17us) server_negotiation.cc:407] Connection header received
0809 09:28:05.142519 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.142531 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.142684 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.142842 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:05.143082 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.144083 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.145688 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.146379 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.146467 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.146852 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.146895 (+ 43us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.147551 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":102,"threads_started":1}
I20260809 09:28:05.147881 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.141409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.9.102.2:35759)
0809 09:28:05.142317 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.142328 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.142350 (+ 22us) server_negotiation.cc:407] Connection header received
0809 09:28:05.142418 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.142424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.143059 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.143206 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:05.143397 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.144324 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.145570 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.146343 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.147352 (+ 1009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.147619 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.147664 (+ 45us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.147751 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":125,"threads_started":1}
I20260809 09:28:05.149969 23085 tablet_service.cc:1467] Processing CreateTablet for tablet c12461445b5947a493178b21961f17b0 (DEFAULT_TABLE table=test-table [id=cb9ea21e1a2b45a7a904fc622b697e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:05.150359 22597 tablet_service.cc:1467] Processing CreateTablet for tablet c12461445b5947a493178b21961f17b0 (DEFAULT_TABLE table=test-table [id=cb9ea21e1a2b45a7a904fc622b697e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:05.150359 22838 tablet_service.cc:1467] Processing CreateTablet for tablet c12461445b5947a493178b21961f17b0 (DEFAULT_TABLE table=test-table [id=cb9ea21e1a2b45a7a904fc622b697e03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:05.150985 23085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12461445b5947a493178b21961f17b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.151274 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12461445b5947a493178b21961f17b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.151274 22597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c12461445b5947a493178b21961f17b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.156961 23232 tablet_bootstrap.cc:492] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f: Bootstrap starting.
I20260809 09:28:05.158541 23235 tablet_bootstrap.cc:492] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79: Bootstrap starting.
I20260809 09:28:05.158828 23233 tablet_bootstrap.cc:492] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584: Bootstrap starting.
I20260809 09:28:05.159504 23232 tablet_bootstrap.cc:654] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.160604 23232 log.cc:824] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f: Log is configured to *not* fsync() on all Append() calls
I20260809 09:28:05.160722 23235 tablet_bootstrap.cc:654] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.161018 23233 tablet_bootstrap.cc:654] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.161530 23235 log.cc:824] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79: Log is configured to *not* fsync() on all Append() calls
I20260809 09:28:05.161777 23233 log.cc:824] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584: Log is configured to *not* fsync() on all Append() calls
I20260809 09:28:05.162649 23232 tablet_bootstrap.cc:492] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f: No bootstrap required, opened a new log
I20260809 09:28:05.162863 23232 ts_tablet_manager.cc:1397] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 09:28:05.163178 23235 tablet_bootstrap.cc:492] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79: No bootstrap required, opened a new log
I20260809 09:28:05.163345 23235 ts_tablet_manager.cc:1397] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 09:28:05.163983 23233 tablet_bootstrap.cc:492] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584: No bootstrap required, opened a new log
I20260809 09:28:05.164189 23233 ts_tablet_manager.cc:1397] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 09:28:05.166013 23235 raft_consensus.cc:348] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.166227 23235 raft_consensus.cc:374] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.166292 23235 raft_consensus.cc:729] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4869fad624c04f58927b0d002df8ce79, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.166683 23235 consensus_queue.cc:260] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.166800 23232 raft_consensus.cc:348] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.167059 23232 raft_consensus.cc:374] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.167140 23232 raft_consensus.cc:729] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90437df78a34b5a854338321873a93f, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.167261 22960 heartbeater.cc:503] Master 127.9.102.62:38761 was elected leader, sending a full tablet report...
I20260809 09:28:05.167528 23235 ts_tablet_manager.cc:1428] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260809 09:28:05.167541 23232 consensus_queue.cc:260] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.167484 23233 raft_consensus.cc:348] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.167687 23233 raft_consensus.cc:374] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.167739 23233 raft_consensus.cc:729] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cfe2644bebf4251930b0be0d02c2584, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.168133 23233 consensus_queue.cc:260] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.168224 23186 heartbeater.cc:503] Master 127.9.102.62:38761 was elected leader, sending a full tablet report...
I20260809 09:28:05.168452 23232 ts_tablet_manager.cc:1428] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 09:28:05.168875 23233 ts_tablet_manager.cc:1428] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:28:05.169274 22720 heartbeater.cc:503] Master 127.9.102.62:38761 was elected leader, sending a full tablet report...
W20260809 09:28:05.196053 22721 tablet.cc:2369] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:28:05.335786 23187 tablet.cc:2369] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:28:05.341145 23242 raft_consensus.cc:482] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:28:05.341332 23242 raft_consensus.cc:504] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.342365 23242 leader_election.cc:288] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759), 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443)
I20260809 09:28:05.346457 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.342814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:50457 (local address 127.9.102.2:35759)
0809 09:28:05.342926 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.342929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.342940 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:28:05.343021 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.343024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.343075 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.343156 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:05.343509 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.344284 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.345437 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.345982 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.346065 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.346113 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.346314 (+ 201us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.346366 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 09:28:05.346479 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.342614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:35759 (local address 127.9.102.3:50457)
0809 09:28:05.342762 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.342778 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.342868 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.343185 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.343187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.343205 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:05.343366 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.343372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.344378 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.344381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.345313 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.345322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.346093 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.346096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.346192 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.346216 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.346287 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.346351 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":28,"spinlock_wait_cycles":23936}
I20260809 09:28:05.347134 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.342614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38443 (local address 127.9.102.3:45669)
0809 09:28:05.343163 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.343177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.343294 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.343536 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.343538 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.343555 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:05.344066 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.344075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.344914 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.344917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.345866 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.345876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.346735 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.346738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.346864 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.346893 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.346989 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.347036 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":176,"spinlock_wait_cycles":768,"thread_start_us":155,"threads_started":1}
I20260809 09:28:05.347167 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.342715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:45669 (local address 127.9.102.1:38443)
0809 09:28:05.342846 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.342850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.343264 (+ 414us) server_negotiation.cc:407] Connection header received
0809 09:28:05.343375 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.343379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.343435 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.343509 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:05.344211 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.344806 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.345988 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.346521 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.346622 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.346693 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.347002 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.347071 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 09:28:05.347182 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12461445b5947a493178b21961f17b0" candidate_uuid: "e90437df78a34b5a854338321873a93f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4869fad624c04f58927b0d002df8ce79" is_pre_election: true
I20260809 09:28:05.347469 22881 raft_consensus.cc:2454] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e90437df78a34b5a854338321873a93f in term 0.
I20260809 09:28:05.347784 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12461445b5947a493178b21961f17b0" candidate_uuid: "e90437df78a34b5a854338321873a93f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cfe2644bebf4251930b0be0d02c2584" is_pre_election: true
I20260809 09:28:05.347980 23003 leader_election.cc:302] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4869fad624c04f58927b0d002df8ce79, e90437df78a34b5a854338321873a93f; no voters:
I20260809 09:28:05.348140 22642 raft_consensus.cc:2454] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e90437df78a34b5a854338321873a93f in term 0.
I20260809 09:28:05.348241 23242 raft_consensus.cc:2781] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:28:05.348320 23242 raft_consensus.cc:482] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:28:05.348387 23242 raft_consensus.cc:3037] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.349407 23242 raft_consensus.cc:504] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.349725 23242 leader_election.cc:288] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [CANDIDATE]: Term 1 election: Requested vote from peers 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759), 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443)
I20260809 09:28:05.349992 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12461445b5947a493178b21961f17b0" candidate_uuid: "e90437df78a34b5a854338321873a93f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4869fad624c04f58927b0d002df8ce79"
I20260809 09:28:05.350069 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c12461445b5947a493178b21961f17b0" candidate_uuid: "e90437df78a34b5a854338321873a93f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cfe2644bebf4251930b0be0d02c2584"
I20260809 09:28:05.350111 22881 raft_consensus.cc:3037] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.350193 22642 raft_consensus.cc:3037] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.351161 22642 raft_consensus.cc:2454] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e90437df78a34b5a854338321873a93f in term 1.
I20260809 09:28:05.351428 22997 leader_election.cc:302] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cfe2644bebf4251930b0be0d02c2584, e90437df78a34b5a854338321873a93f; no voters:
I20260809 09:28:05.351513 22881 raft_consensus.cc:2454] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e90437df78a34b5a854338321873a93f in term 1.
I20260809 09:28:05.351595 23242 raft_consensus.cc:2781] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:28:05.351794 23242 raft_consensus.cc:686] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 1 LEADER]: Becoming Leader. State: Replica: e90437df78a34b5a854338321873a93f, State: Running, Role: LEADER
I20260809 09:28:05.352121 23242 consensus_queue.cc:237] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.355037 22351 catalog_manager.cc:5495] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f reported cstate change: term changed from 0 to 1, leader changed from <none> to e90437df78a34b5a854338321873a93f (127.9.102.3). New cstate: current_term: 1 leader_uuid: "e90437df78a34b5a854338321873a93f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } health_report { overall_health: HEALTHY } } }
I20260809 09:28:05.385883 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.382422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36650 (local address 127.9.102.3:35891)
0809 09:28:05.382552 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.382558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.382576 (+ 18us) server_negotiation.cc:407] Connection header received
0809 09:28:05.382774 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.382777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.382843 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.382953 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:28:05.383198 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.383674 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.384568 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.384775 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.384887 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.385156 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.385514 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.385642 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.385697 (+ 55us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.385745 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 09:28:05.388852 22961 tablet.cc:2369] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:28:05.391144 22881 raft_consensus.cc:1261] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Refusing update from remote peer e90437df78a34b5a854338321873a93f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:05.391146 22642 raft_consensus.cc:1261] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Refusing update from remote peer e90437df78a34b5a854338321873a93f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:05.391785 23245 consensus_queue.cc:1035] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:05.392009 23242 consensus_queue.cc:1035] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:05.402463 23251 mvcc.cc:205] Tried to move back new op lower bound from 7316552439355699200 to 7316552439204790272. Current Snapshot: MvccSnapshot[applied={T|T < 7316552439355699200 or (T in {7316552439355699200})}]
I20260809 09:28:05.402652 23254 mvcc.cc:205] Tried to move back new op lower bound from 7316552439355699200 to 7316552439204790272. Current Snapshot: MvccSnapshot[applied={T|T < 7316552439355699200 or (T in {7316552439355699200})}]
I20260809 09:28:05.407204 22352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:28:05.407356 22352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:28:05.409535 22352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f: Sending DeleteTablet for 3 replicas of tablet c12461445b5947a493178b21961f17b0
I20260809 09:28:05.410271 23085 tablet_service.cc:1514] Processing DeleteTablet for tablet c12461445b5947a493178b21961f17b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:05 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:36642
I20260809 09:28:05.410266 22838 tablet_service.cc:1514] Processing DeleteTablet for tablet c12461445b5947a493178b21961f17b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:05 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:50102
I20260809 09:28:05.410256 22597 tablet_service.cc:1514] Processing DeleteTablet for tablet c12461445b5947a493178b21961f17b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:05 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:54802
I20260809 09:28:05.410686 23268 tablet_replica.cc:330] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584: stopping tablet replica
I20260809 09:28:05.410703 23266 tablet_replica.cc:330] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79: stopping tablet replica
I20260809 09:28:05.410952 23266 raft_consensus.cc:2229] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:28:05.410975 23268 raft_consensus.cc:2229] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:28:05.411268 23268 raft_consensus.cc:2258] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:05.411269 23266 raft_consensus.cc:2258] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:05.411378 22352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:28:05.411762 22352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:28:05.412621 23266 ts_tablet_manager.cc:1905] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:05.412701 23268 ts_tablet_manager.cc:1905] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:05.415331 23266 ts_tablet_manager.cc:1918] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:28:05.415496 23266 log.cc:1196] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/c12461445b5947a493178b21961f17b0
I20260809 09:28:05.415488 23268 ts_tablet_manager.cc:1918] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:28:05.415577 23268 log.cc:1196] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/wal/wals/c12461445b5947a493178b21961f17b0
I20260809 09:28:05.415910 23266 ts_tablet_manager.cc:1939] T c12461445b5947a493178b21961f17b0 P 4869fad624c04f58927b0d002df8ce79: Deleting consensus metadata
I20260809 09:28:05.416030 23268 ts_tablet_manager.cc:1939] T c12461445b5947a493178b21961f17b0 P 2cfe2644bebf4251930b0be0d02c2584: Deleting consensus metadata
I20260809 09:28:05.416915 22597 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4be027c80343679283ae115ff6c101 (DEFAULT_TABLE table=test-table [id=c0000ee1c11449ab830d55ad34e70ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:05.416936 22838 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4be027c80343679283ae115ff6c101 (DEFAULT_TABLE table=test-table [id=c0000ee1c11449ab830d55ad34e70ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:05.417158 22597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4be027c80343679283ae115ff6c101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.417173 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4be027c80343679283ae115ff6c101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.417222 22333 catalog_manager.cc:4841] TS 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443): tablet c12461445b5947a493178b21961f17b0 (table test-table [id=cb9ea21e1a2b45a7a904fc622b697e03]) successfully deleted
I20260809 09:28:05.418565 23233 tablet_bootstrap.cc:492] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584: Bootstrap starting.
I20260809 09:28:05.418717 22338 catalog_manager.cc:4841] TS 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759): tablet c12461445b5947a493178b21961f17b0 (table test-table [id=cb9ea21e1a2b45a7a904fc622b697e03]) successfully deleted
I20260809 09:28:05.418787 23235 tablet_bootstrap.cc:492] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79: Bootstrap starting.
I20260809 09:28:05.419224 23085 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4be027c80343679283ae115ff6c101 (DEFAULT_TABLE table=test-table [id=c0000ee1c11449ab830d55ad34e70ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:05.419404 23233 tablet_bootstrap.cc:654] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.419459 23085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4be027c80343679283ae115ff6c101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.419747 23235 tablet_bootstrap.cc:654] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.420951 23233 tablet_bootstrap.cc:492] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584: No bootstrap required, opened a new log
I20260809 09:28:05.421036 23233 ts_tablet_manager.cc:1397] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260809 09:28:05.421262 23235 tablet_bootstrap.cc:492] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79: No bootstrap required, opened a new log
I20260809 09:28:05.421339 23235 ts_tablet_manager.cc:1397] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260809 09:28:05.421442 23233 raft_consensus.cc:348] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.421533 23233 raft_consensus.cc:374] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.421562 23233 raft_consensus.cc:729] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cfe2644bebf4251930b0be0d02c2584, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.421672 23233 consensus_queue.cc:260] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.421741 23235 raft_consensus.cc:348] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.421857 23235 raft_consensus.cc:374] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.421891 23235 raft_consensus.cc:729] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4869fad624c04f58927b0d002df8ce79, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.421907 23233 ts_tablet_manager.cc:1428] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 09:28:05.422006 23235 consensus_queue.cc:260] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.422262 23235 ts_tablet_manager.cc:1428] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 09:28:05.423918 23232 tablet_bootstrap.cc:492] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f: Bootstrap starting.
I20260809 09:28:05.424307 23269 tablet_replica.cc:330] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f: stopping tablet replica
I20260809 09:28:05.424633 23269 raft_consensus.cc:2229] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:28:05.424677 23232 tablet_bootstrap.cc:654] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.425046 23269 raft_consensus.cc:2258] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:05.426056 23232 tablet_bootstrap.cc:492] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f: No bootstrap required, opened a new log
I20260809 09:28:05.426182 23232 ts_tablet_manager.cc:1397] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:28:05.426504 23269 ts_tablet_manager.cc:1905] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:05.426576 23232 raft_consensus.cc:348] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.426671 23232 raft_consensus.cc:374] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.426705 23232 raft_consensus.cc:729] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90437df78a34b5a854338321873a93f, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.426810 23232 consensus_queue.cc:260] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.427084 23232 ts_tablet_manager.cc:1428] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:28:05.428484 23269 ts_tablet_manager.cc:1918] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:28:05.428597 23269 log.cc:1196] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/c12461445b5947a493178b21961f17b0
I20260809 09:28:05.428887 23269 ts_tablet_manager.cc:1939] T c12461445b5947a493178b21961f17b0 P e90437df78a34b5a854338321873a93f: Deleting consensus metadata
I20260809 09:28:05.429741 22338 catalog_manager.cc:4841] TS e90437df78a34b5a854338321873a93f (127.9.102.3:35891): tablet c12461445b5947a493178b21961f17b0 (table test-table [id=cb9ea21e1a2b45a7a904fc622b697e03]) successfully deleted
I20260809 09:28:05.449707 23243 raft_consensus.cc:482] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:28:05.449817 23243 raft_consensus.cc:504] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.450845 23243 leader_election.cc:288] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759), e90437df78a34b5a854338321873a93f (127.9.102.3:35891)
I20260809 09:28:05.455375 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.451540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:47253 (local address 127.9.102.3:35891)
0809 09:28:05.452020 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.452033 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.452045 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:28:05.452118 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.452120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.452161 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.452251 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:05.452714 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.453272 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.454241 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.454751 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.454862 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.454920 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.455210 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.455277 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390}
I20260809 09:28:05.455386 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.451142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:33753 (local address 127.9.102.2:35759)
0809 09:28:05.451685 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.451689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.451698 (+ 9us) server_negotiation.cc:407] Connection header received
0809 09:28:05.451734 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.451736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.451766 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.451853 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:05.452166 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.452872 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.454308 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.454808 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.454883 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.454924 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.455238 (+ 314us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.455301 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439}
I20260809 09:28:05.455389 23279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.451360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:35891 (local address 127.9.102.1:47253)
0809 09:28:05.451698 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.451712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.451802 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.452260 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.452263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.452279 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:05.452586 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.452595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.453382 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.453385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.454104 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.454112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.454903 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.454907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.455063 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.455091 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.455171 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.455235 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":25,"thread_start_us":115,"threads_started":1}
I20260809 09:28:05.455835 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.451029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:35759 (local address 127.9.102.1:33753)
0809 09:28:05.451233 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.451248 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.451355 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.451864 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.451867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.451884 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:05.452003 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.452026 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.453012 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.453016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.454191 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.454203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.454979 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.454982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.455089 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.455106 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.455644 (+ 538us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.455705 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":18,"thread_start_us":80,"threads_started":1}
I20260809 09:28:05.456171 23113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4be027c80343679283ae115ff6c101" candidate_uuid: "2cfe2644bebf4251930b0be0d02c2584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e90437df78a34b5a854338321873a93f" is_pre_election: true
I20260809 09:28:05.456238 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4be027c80343679283ae115ff6c101" candidate_uuid: "2cfe2644bebf4251930b0be0d02c2584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4869fad624c04f58927b0d002df8ce79" is_pre_election: true
I20260809 09:28:05.456360 22881 raft_consensus.cc:2454] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cfe2644bebf4251930b0be0d02c2584 in term 0.
I20260809 09:28:05.456373 23113 raft_consensus.cc:2454] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cfe2644bebf4251930b0be0d02c2584 in term 0.
I20260809 09:28:05.456741 22517 leader_election.cc:302] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cfe2644bebf4251930b0be0d02c2584, 4869fad624c04f58927b0d002df8ce79; no voters:
I20260809 09:28:05.456982 23243 raft_consensus.cc:2781] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:28:05.457050 23243 raft_consensus.cc:482] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:28:05.457089 23243 raft_consensus.cc:3037] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.458104 23243 raft_consensus.cc:504] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.458425 23243 leader_election.cc:288] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [CANDIDATE]: Term 1 election: Requested vote from peers 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759), e90437df78a34b5a854338321873a93f (127.9.102.3:35891)
I20260809 09:28:05.458707 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4be027c80343679283ae115ff6c101" candidate_uuid: "2cfe2644bebf4251930b0be0d02c2584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4869fad624c04f58927b0d002df8ce79"
I20260809 09:28:05.458845 22881 raft_consensus.cc:3037] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.458838 23113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4be027c80343679283ae115ff6c101" candidate_uuid: "2cfe2644bebf4251930b0be0d02c2584" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e90437df78a34b5a854338321873a93f"
I20260809 09:28:05.458936 23113 raft_consensus.cc:3037] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.459949 22881 raft_consensus.cc:2454] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cfe2644bebf4251930b0be0d02c2584 in term 1.
I20260809 09:28:05.460041 23113 raft_consensus.cc:2454] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cfe2644bebf4251930b0be0d02c2584 in term 1.
I20260809 09:28:05.460274 22517 leader_election.cc:302] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cfe2644bebf4251930b0be0d02c2584, 4869fad624c04f58927b0d002df8ce79; no voters:
I20260809 09:28:05.460430 23243 raft_consensus.cc:2781] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:28:05.460630 23243 raft_consensus.cc:686] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 LEADER]: Becoming Leader. State: Replica: 2cfe2644bebf4251930b0be0d02c2584, State: Running, Role: LEADER
I20260809 09:28:05.460902 23243 consensus_queue.cc:237] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.462813 22352 catalog_manager.cc:5495] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1). New cstate: current_term: 1 leader_uuid: "2cfe2644bebf4251930b0be0d02c2584" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
Aug 09 09:28:05 dist-test-slave-7vjv krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267684, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.62@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:28:05.499063 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.490254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38761 (local address 127.0.0.1:36936)
0809 09:28:05.490661 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.490672 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.490840 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.491146 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.491148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.491546 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:05.491643 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.491649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.492707 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.492709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.493214 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.493222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.493762 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.493764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.493848 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.494443 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:05.494453 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:05.495956 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:05.497678 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:05.497686 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:05.497691 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:05.497888 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:05.498166 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:05.498168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:05.498169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:05.498239 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:05.498652 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:05.498657 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:05.498748 (+ 91us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.498832 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.498928 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":16,"thread_start_us":119,"threads_started":1}
I20260809 09:28:05.499234 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.490331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.9.102.62:38761)
0809 09:28:05.490468 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.490491 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.490769 (+ 278us) server_negotiation.cc:407] Connection header received
0809 09:28:05.490957 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.490961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.491014 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.491116 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:05.491778 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.492592 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.493334 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.493644 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.493747 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.496112 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:05.496117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:05.496121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:05.496147 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:05.497570 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:05.497983 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:05.497986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:05.497988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:05.498075 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:05.498325 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:05.498328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:05.498330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:05.498547 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:05.498651 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.498934 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.499049 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":59}
I20260809 09:28:05.503916 22352 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:36936:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 09:28:05.504642 22352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:28:05.509510 22838 tablet_service.cc:1467] Processing CreateTablet for tablet b2c3d763c32141bd89a639154f831d27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd1d35699b8a41c284aead38e2f492a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 09:28:05.509510 22597 tablet_service.cc:1467] Processing CreateTablet for tablet b2c3d763c32141bd89a639154f831d27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd1d35699b8a41c284aead38e2f492a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 09:28:05.509908 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c3d763c32141bd89a639154f831d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.509908 22597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c3d763c32141bd89a639154f831d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.512670 23235 tablet_bootstrap.cc:492] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: Bootstrap starting.
I20260809 09:28:05.512871 23085 tablet_service.cc:1467] Processing CreateTablet for tablet b2c3d763c32141bd89a639154f831d27 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dd1d35699b8a41c284aead38e2f492a4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 09:28:05.513213 23085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c3d763c32141bd89a639154f831d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.513602 23235 tablet_bootstrap.cc:654] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.515796 23232 tablet_bootstrap.cc:492] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f: Bootstrap starting.
I20260809 09:28:05.515919 23233 tablet_bootstrap.cc:492] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584: Bootstrap starting.
I20260809 09:28:05.516757 23232 tablet_bootstrap.cc:654] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.516799 23233 tablet_bootstrap.cc:654] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.518066 23233 tablet_bootstrap.cc:492] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584: No bootstrap required, opened a new log
I20260809 09:28:05.518175 23233 ts_tablet_manager.cc:1397] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:28:05.518364 23232 tablet_bootstrap.cc:492] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f: No bootstrap required, opened a new log
I20260809 09:28:05.518468 23232 ts_tablet_manager.cc:1397] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 09:28:05.518635 23233 raft_consensus.cc:348] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.518750 23233 raft_consensus.cc:374] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.518788 23233 raft_consensus.cc:729] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cfe2644bebf4251930b0be0d02c2584, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.518918 23233 consensus_queue.cc:260] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.518939 23232 raft_consensus.cc:348] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.519079 23232 raft_consensus.cc:374] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.519136 23232 raft_consensus.cc:729] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90437df78a34b5a854338321873a93f, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.519191 23233 ts_tablet_manager.cc:1428] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:28:05.519284 23232 consensus_queue.cc:260] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.519371 23243 tablet_replica.cc:439] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } } }
I20260809 09:28:05.519476 23243 tablet_replica.cc:442] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:28:05.519780 23245 tablet_replica.cc:439] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } } }
I20260809 09:28:05.519930 23245 tablet_replica.cc:442] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:28:05.520336 23235 tablet_bootstrap.cc:492] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: No bootstrap required, opened a new log
I20260809 09:28:05.520395 23235 ts_tablet_manager.cc:1397] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260809 09:28:05.520875 23235 raft_consensus.cc:348] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.520947 23235 raft_consensus.cc:374] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.520965 23235 raft_consensus.cc:729] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4869fad624c04f58927b0d002df8ce79, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.521082 23235 consensus_queue.cc:260] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.521320 23235 ts_tablet_manager.cc:1428] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260809 09:28:05.521292 23241 tablet_replica.cc:439] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } } }
I20260809 09:28:05.521404 23241 tablet_replica.cc:442] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:28:05.522393 23232 ts_tablet_manager.cc:1428] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 09:28:05.575964 23241 raft_consensus.cc:482] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:28:05.576155 23241 raft_consensus.cc:504] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.577256 23241 leader_election.cc:288] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443), e90437df78a34b5a854338321873a93f (127.9.102.3:35891)
I20260809 09:28:05.581153 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.577432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38443 (local address 127.9.102.2:41127)
0809 09:28:05.577726 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.577741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.577843 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.578110 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.578113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.578130 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:05.578271 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.578278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.579080 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.579083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.579958 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.579968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.580671 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.580675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.580808 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.580850 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.580916 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.580990 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":52,"thread_start_us":99,"threads_started":1}
I20260809 09:28:05.581179 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.577488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:41127 (local address 127.9.102.1:38443)
0809 09:28:05.577661 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.577666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.577874 (+ 208us) server_negotiation.cc:407] Connection header received
0809 09:28:05.577949 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.577953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.578003 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.578081 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:05.578395 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.578932 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.580127 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.580547 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.580639 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.580692 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.580956 (+ 264us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.581032 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 09:28:05.581377 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.577616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:51541 (local address 127.9.102.3:35891)
0809 09:28:05.577726 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.577731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.577978 (+ 247us) server_negotiation.cc:407] Connection header received
0809 09:28:05.578112 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.578115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.578155 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.578231 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:05.578555 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.578964 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.580165 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.580734 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.580863 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.580934 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.581172 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.581258 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:28:05.581496 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.577479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:35891 (local address 127.9.102.2:51541)
0809 09:28:05.577868 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.577885 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.577992 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.578269 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.578271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.578286 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:05.578433 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.578440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.579132 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.579134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.579967 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.579976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.580918 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.580923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.581020 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.581045 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.581127 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.581184 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":37,"thread_start_us":112,"threads_started":1}
I20260809 09:28:05.581738 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c3d763c32141bd89a639154f831d27" candidate_uuid: "4869fad624c04f58927b0d002df8ce79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cfe2644bebf4251930b0be0d02c2584" is_pre_election: true
I20260809 09:28:05.581892 22642 raft_consensus.cc:2454] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4869fad624c04f58927b0d002df8ce79 in term 0.
I20260809 09:28:05.581938 23113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c3d763c32141bd89a639154f831d27" candidate_uuid: "4869fad624c04f58927b0d002df8ce79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e90437df78a34b5a854338321873a93f" is_pre_election: true
I20260809 09:28:05.582062 23113 raft_consensus.cc:2454] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4869fad624c04f58927b0d002df8ce79 in term 0.
I20260809 09:28:05.582259 22765 leader_election.cc:302] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cfe2644bebf4251930b0be0d02c2584, 4869fad624c04f58927b0d002df8ce79; no voters:
I20260809 09:28:05.582526 23241 raft_consensus.cc:2781] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:28:05.582595 23241 raft_consensus.cc:482] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:28:05.582626 23241 raft_consensus.cc:3037] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.583611 23241 raft_consensus.cc:504] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.583971 23241 leader_election.cc:288] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [CANDIDATE]: Term 1 election: Requested vote from peers 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443), e90437df78a34b5a854338321873a93f (127.9.102.3:35891)
I20260809 09:28:05.584348 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c3d763c32141bd89a639154f831d27" candidate_uuid: "4869fad624c04f58927b0d002df8ce79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cfe2644bebf4251930b0be0d02c2584"
I20260809 09:28:05.584340 23113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c3d763c32141bd89a639154f831d27" candidate_uuid: "4869fad624c04f58927b0d002df8ce79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e90437df78a34b5a854338321873a93f"
I20260809 09:28:05.584466 22642 raft_consensus.cc:3037] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.584481 23113 raft_consensus.cc:3037] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.585565 22642 raft_consensus.cc:2454] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4869fad624c04f58927b0d002df8ce79 in term 1.
I20260809 09:28:05.585565 23113 raft_consensus.cc:2454] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4869fad624c04f58927b0d002df8ce79 in term 1.
I20260809 09:28:05.585830 22765 leader_election.cc:302] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cfe2644bebf4251930b0be0d02c2584, 4869fad624c04f58927b0d002df8ce79; no voters:
I20260809 09:28:05.585996 23241 raft_consensus.cc:2781] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:28:05.586246 23241 raft_consensus.cc:686] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [term 1 LEADER]: Becoming Leader. State: Replica: 4869fad624c04f58927b0d002df8ce79, State: Running, Role: LEADER
I20260809 09:28:05.586515 23241 consensus_queue.cc:237] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.587646 23302 tablet_replica.cc:439] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: TxnStatusTablet state changed. Reason: New leader 4869fad624c04f58927b0d002df8ce79. Latest consensus state: current_term: 1 leader_uuid: "4869fad624c04f58927b0d002df8ce79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } } }
I20260809 09:28:05.587755 23302 tablet_replica.cc:442] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: This TxnStatusTablet replica's current role is: LEADER
I20260809 09:28:05.587989 23306 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 09:28:05.588184 23306 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 09:28:05.588814 22352 catalog_manager.cc:5495] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4869fad624c04f58927b0d002df8ce79 (127.9.102.2). New cstate: current_term: 1 leader_uuid: "4869fad624c04f58927b0d002df8ce79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260809 09:28:05.617799 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.614173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:35759 (local address 127.0.0.1:50118)
0809 09:28:05.614331 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.614349 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.614425 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.614762 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.614763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.614769 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 09:28:05.614848 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.614852 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.615765 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.615767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.616367 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.616376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.616855 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.616857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.616909 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.616946 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.617593 (+ 647us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.617604 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.617650 (+ 46us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.617710 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 09:28:05.617830 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.614263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.9.102.2:35759)
0809 09:28:05.614368 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.614372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.614386 (+ 14us) server_negotiation.cc:407] Connection header received
0809 09:28:05.614577 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.614580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.614634 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.614730 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:28:05.614942 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.615671 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.616465 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.616760 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.616850 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.617072 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.617506 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.617606 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.617660 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.617704 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 09:28:05.620532 22642 raft_consensus.cc:1261] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Refusing update from remote peer 4869fad624c04f58927b0d002df8ce79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:05.620633 23113 raft_consensus.cc:1261] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Refusing update from remote peer 4869fad624c04f58927b0d002df8ce79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:05.621117 23241 consensus_queue.cc:1035] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:05.621294 23302 consensus_queue.cc:1035] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:05.622566 23245 tablet_replica.cc:439] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f: TxnStatusTablet state changed. Reason: New leader 4869fad624c04f58927b0d002df8ce79. Latest consensus state: current_term: 1 leader_uuid: "4869fad624c04f58927b0d002df8ce79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } } }
I20260809 09:28:05.622682 23245 tablet_replica.cc:442] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:28:05.622615 23243 tablet_replica.cc:439] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584: TxnStatusTablet state changed. Reason: New leader 4869fad624c04f58927b0d002df8ce79. Latest consensus state: current_term: 1 leader_uuid: "4869fad624c04f58927b0d002df8ce79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } } }
I20260809 09:28:05.622727 23243 tablet_replica.cc:442] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:28:05.625761 23302 tablet_replica.cc:439] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4869fad624c04f58927b0d002df8ce79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } } }
I20260809 09:28:05.625857 23302 tablet_replica.cc:442] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: This TxnStatusTablet replica's current role is: LEADER
I20260809 09:28:05.625821 23241 tablet_replica.cc:439] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4869fad624c04f58927b0d002df8ce79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } } }
I20260809 09:28:05.625895 23241 tablet_replica.cc:442] T b2c3d763c32141bd89a639154f831d27 P 4869fad624c04f58927b0d002df8ce79: This TxnStatusTablet replica's current role is: LEADER
I20260809 09:28:05.627239 23248 mvcc.cc:205] Tried to move back new op lower bound from 7316552440296263680 to 7316552440164311040. Current Snapshot: MvccSnapshot[applied={T|T < 7316552440296263680 or (T in {7316552440296263680})}]
I20260809 09:28:05.627298 23243 tablet_replica.cc:439] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4869fad624c04f58927b0d002df8ce79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } } }
I20260809 09:28:05.627395 23243 tablet_replica.cc:442] T b2c3d763c32141bd89a639154f831d27 P 2cfe2644bebf4251930b0be0d02c2584: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:28:05.628065 23245 tablet_replica.cc:439] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4869fad624c04f58927b0d002df8ce79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } } }
I20260809 09:28:05.628165 23245 tablet_replica.cc:442] T b2c3d763c32141bd89a639154f831d27 P e90437df78a34b5a854338321873a93f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 09:28:05.635193 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.631771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.9.102.1:38443)
0809 09:28:05.631854 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.631859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.631908 (+ 49us) server_negotiation.cc:407] Connection header received
0809 09:28:05.632082 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.632084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.632156 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.632263 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:28:05.632477 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.632950 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.633773 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.634003 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.634151 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.634400 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.634747 (+ 347us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.634879 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.634964 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.635030 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 09:28:05.645015 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.641884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:43263 (local address 127.9.102.2:35759)
0809 09:28:05.642011 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.642015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.642178 (+ 163us) server_negotiation.cc:407] Connection header received
0809 09:28:05.642261 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.642262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.642296 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.642350 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:28:05.642607 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.643012 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.644132 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.644316 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.644387 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.644684 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.644746 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.644792 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.644905 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.644940 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 09:28:05.645215 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.641822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:35759 (local address 127.9.102.1:43263)
0809 09:28:05.642089 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.642102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.642193 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.642391 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.642393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.642400 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 09:28:05.642507 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.642512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.643109 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.643112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.643981 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.643991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.644420 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.644423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.644520 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.644564 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.644821 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.644826 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.644882 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.644944 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":103,"threads_started":1}
I20260809 09:28:05.651594 23113 raft_consensus.cc:1261] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Refusing update from remote peer 2cfe2644bebf4251930b0be0d02c2584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:05.651911 22881 raft_consensus.cc:1261] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Refusing update from remote peer 2cfe2644bebf4251930b0be0d02c2584: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:05.652375 23280 consensus_queue.cc:1035] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:05.652595 23243 consensus_queue.cc:1035] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:05.673424 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.669831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:50943 (local address 127.9.102.1:38443)
0809 09:28:05.669949 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.669953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.670130 (+ 177us) server_negotiation.cc:407] Connection header received
0809 09:28:05.670227 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.670229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.670261 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.670316 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:28:05.670593 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.671146 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.672374 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.672559 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.672710 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.673022 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.673089 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.673156 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.673278 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.673320 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:28:05.673456 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.669768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38443 (local address 127.9.102.2:50943)
0809 09:28:05.670049 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.670062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.670154 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.670351 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.670354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.670360 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 09:28:05.670473 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.670479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.671254 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.671257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.672253 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.672265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.672753 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.672756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.672848 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.672893 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.673190 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.673196 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.673260 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.673328 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":115,"threads_started":1}
I20260809 09:28:05.693478 22352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:28:05.693630 22352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:28:05.695699 22352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f: Sending DeleteTablet for 3 replicas of tablet 3c4be027c80343679283ae115ff6c101
I20260809 09:28:05.696617 22597 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4be027c80343679283ae115ff6c101 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:05 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:54802
I20260809 09:28:05.700343 23268 tablet_replica.cc:330] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584: stopping tablet replica
I20260809 09:28:05.700343 23085 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4be027c80343679283ae115ff6c101 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:05 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:36642
I20260809 09:28:05.700465 23268 raft_consensus.cc:2229] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:28:05.700500 23269 tablet_replica.cc:330] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f: stopping tablet replica
I20260809 09:28:05.700598 23269 raft_consensus.cc:2229] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:28:05.700693 23269 raft_consensus.cc:2258] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:05.700716 23268 raft_consensus.cc:2258] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:05.701718 23269 ts_tablet_manager.cc:1905] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:05.701891 23268 ts_tablet_manager.cc:1905] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:05.704275 23269 ts_tablet_manager.cc:1918] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:28:05.704351 23269 log.cc:1196] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/3c4be027c80343679283ae115ff6c101
I20260809 09:28:05.704589 23269 ts_tablet_manager.cc:1939] T 3c4be027c80343679283ae115ff6c101 P e90437df78a34b5a854338321873a93f: Deleting consensus metadata
I20260809 09:28:05.705292 22338 catalog_manager.cc:4841] TS e90437df78a34b5a854338321873a93f (127.9.102.3:35891): tablet 3c4be027c80343679283ae115ff6c101 (table test-table [id=c0000ee1c11449ab830d55ad34e70ccb]) successfully deleted
I20260809 09:28:05.708140 22838 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4be027c80343679283ae115ff6c101 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:05 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:50102
I20260809 09:28:05.708240 23268 ts_tablet_manager.cc:1918] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:28:05.708307 23268 log.cc:1196] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/wal/wals/3c4be027c80343679283ae115ff6c101
I20260809 09:28:05.708313 23266 tablet_replica.cc:330] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79: stopping tablet replica
I20260809 09:28:05.708408 23266 raft_consensus.cc:2229] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:28:05.708511 23266 raft_consensus.cc:2258] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:05.708774 23268 ts_tablet_manager.cc:1939] T 3c4be027c80343679283ae115ff6c101 P 2cfe2644bebf4251930b0be0d02c2584: Deleting consensus metadata
I20260809 09:28:05.709197 23266 ts_tablet_manager.cc:1905] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:05.709373 22333 catalog_manager.cc:4841] TS 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443): tablet 3c4be027c80343679283ae115ff6c101 (table test-table [id=c0000ee1c11449ab830d55ad34e70ccb]) successfully deleted
I20260809 09:28:05.711541 23266 ts_tablet_manager.cc:1918] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:28:05.711615 23266 log.cc:1196] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/3c4be027c80343679283ae115ff6c101
I20260809 09:28:05.711891 23266 ts_tablet_manager.cc:1939] T 3c4be027c80343679283ae115ff6c101 P 4869fad624c04f58927b0d002df8ce79: Deleting consensus metadata
I20260809 09:28:05.713141 22338 catalog_manager.cc:4841] TS 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759): tablet 3c4be027c80343679283ae115ff6c101 (table test-table [id=c0000ee1c11449ab830d55ad34e70ccb]) successfully deleted
Aug 09 09:28:05 dist-test-slave-7vjv krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 09:28:05 dist-test-slave-7vjv krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:05 dist-test-slave-7vjv krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 09:28:05.737134 22449 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:05.731054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36938 (local address 127.9.102.62:38761)
0809 09:28:05.731173 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.731177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.731413 (+ 236us) server_negotiation.cc:407] Connection header received
0809 09:28:05.731559 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.731563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.731611 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.731700 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:05.732428 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.732907 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.733816 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.734028 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.734149 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.737040 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36938: BlockingRecv error: recv got EOF from 127.0.0.1:36938 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 09:28:05 dist-test-slave-7vjv krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM
I20260809 09:28:05.746881 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.738878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36954 (local address 127.9.102.62:38761)
0809 09:28:05.738976 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.738981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.739369 (+ 388us) server_negotiation.cc:407] Connection header received
0809 09:28:05.739475 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.739478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.739514 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.739623 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:05.740202 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.740664 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.741408 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.741606 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.741708 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.743862 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:05.743866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:05.743868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:05.743897 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:05.745249 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:05.745649 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:05.745653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:05.745655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:05.745732 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:05.746029 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:05.746032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:05.746034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:05.746212 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:05.746351 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.746576 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.746676 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 09:28:05.749367 22352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:28:05.749785 22352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:28:05.754241 22597 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8d14023a5e43229cc115f32a4cd212 (DEFAULT_TABLE table=test-table [id=ca81a786a56c4a53b765bafb1b2cdd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:05.754608 22597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8d14023a5e43229cc115f32a4cd212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.754752 22838 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8d14023a5e43229cc115f32a4cd212 (DEFAULT_TABLE table=test-table [id=ca81a786a56c4a53b765bafb1b2cdd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:05.754998 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8d14023a5e43229cc115f32a4cd212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.757200 23235 tablet_bootstrap.cc:492] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79: Bootstrap starting.
I20260809 09:28:05.757248 23233 tablet_bootstrap.cc:492] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584: Bootstrap starting.
I20260809 09:28:05.757414 23085 tablet_service.cc:1467] Processing CreateTablet for tablet 2d8d14023a5e43229cc115f32a4cd212 (DEFAULT_TABLE table=test-table [id=ca81a786a56c4a53b765bafb1b2cdd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:05.757684 23085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d8d14023a5e43229cc115f32a4cd212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.758013 23235 tablet_bootstrap.cc:654] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.758116 23233 tablet_bootstrap.cc:654] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.760259 23232 tablet_bootstrap.cc:492] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f: Bootstrap starting.
I20260809 09:28:05.760252 23233 tablet_bootstrap.cc:492] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584: No bootstrap required, opened a new log
I20260809 09:28:05.760336 23233 ts_tablet_manager.cc:1397] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260809 09:28:05.760772 23233 raft_consensus.cc:348] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.760886 23233 raft_consensus.cc:374] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.760917 23233 raft_consensus.cc:729] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cfe2644bebf4251930b0be0d02c2584, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.761104 23232 tablet_bootstrap.cc:654] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.761034 23233 consensus_queue.cc:260] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.761324 23233 ts_tablet_manager.cc:1428] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 09:28:05.763164 23235 tablet_bootstrap.cc:492] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79: No bootstrap required, opened a new log
I20260809 09:28:05.763259 23235 ts_tablet_manager.cc:1397] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260809 09:28:05.763689 23235 raft_consensus.cc:348] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.763782 23235 raft_consensus.cc:374] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.763816 23235 raft_consensus.cc:729] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4869fad624c04f58927b0d002df8ce79, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.763921 23232 tablet_bootstrap.cc:492] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f: No bootstrap required, opened a new log
I20260809 09:28:05.763934 23235 consensus_queue.cc:260] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.763985 23232 ts_tablet_manager.cc:1397] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260809 09:28:05.764178 23235 ts_tablet_manager.cc:1428] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:28:05.764386 23232 raft_consensus.cc:348] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.764487 23232 raft_consensus.cc:374] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.764518 23232 raft_consensus.cc:729] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90437df78a34b5a854338321873a93f, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.764638 23232 consensus_queue.cc:260] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.764875 23232 ts_tablet_manager.cc:1428] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:28:05.803208 23245 raft_consensus.cc:482] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:28:05.803293 23245 raft_consensus.cc:504] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.803673 23245 leader_election.cc:288] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443), 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759)
I20260809 09:28:05.804014 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8d14023a5e43229cc115f32a4cd212" candidate_uuid: "e90437df78a34b5a854338321873a93f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cfe2644bebf4251930b0be0d02c2584" is_pre_election: true
I20260809 09:28:05.804160 22642 raft_consensus.cc:2454] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e90437df78a34b5a854338321873a93f in term 0.
I20260809 09:28:05.804234 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8d14023a5e43229cc115f32a4cd212" candidate_uuid: "e90437df78a34b5a854338321873a93f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4869fad624c04f58927b0d002df8ce79" is_pre_election: true
I20260809 09:28:05.804382 22881 raft_consensus.cc:2454] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e90437df78a34b5a854338321873a93f in term 0.
I20260809 09:28:05.804487 22997 leader_election.cc:302] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cfe2644bebf4251930b0be0d02c2584, e90437df78a34b5a854338321873a93f; no voters:
I20260809 09:28:05.804641 23245 raft_consensus.cc:2781] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:28:05.804695 23245 raft_consensus.cc:482] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:28:05.804740 23245 raft_consensus.cc:3037] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.805611 23245 raft_consensus.cc:504] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.805959 23245 leader_election.cc:288] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [CANDIDATE]: Term 1 election: Requested vote from peers 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443), 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759)
I20260809 09:28:05.806221 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8d14023a5e43229cc115f32a4cd212" candidate_uuid: "e90437df78a34b5a854338321873a93f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cfe2644bebf4251930b0be0d02c2584"
I20260809 09:28:05.806383 22642 raft_consensus.cc:3037] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.806306 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d8d14023a5e43229cc115f32a4cd212" candidate_uuid: "e90437df78a34b5a854338321873a93f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4869fad624c04f58927b0d002df8ce79"
I20260809 09:28:05.806501 22881 raft_consensus.cc:3037] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.807349 22881 raft_consensus.cc:2454] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e90437df78a34b5a854338321873a93f in term 1.
I20260809 09:28:05.807484 22642 raft_consensus.cc:2454] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e90437df78a34b5a854338321873a93f in term 1.
I20260809 09:28:05.807673 23003 leader_election.cc:302] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4869fad624c04f58927b0d002df8ce79, e90437df78a34b5a854338321873a93f; no voters:
I20260809 09:28:05.807896 23245 raft_consensus.cc:2781] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:28:05.808012 23245 raft_consensus.cc:686] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 1 LEADER]: Becoming Leader. State: Replica: e90437df78a34b5a854338321873a93f, State: Running, Role: LEADER
I20260809 09:28:05.808151 23245 consensus_queue.cc:237] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.809718 22352 catalog_manager.cc:5495] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f reported cstate change: term changed from 0 to 1, leader changed from <none> to e90437df78a34b5a854338321873a93f (127.9.102.3). New cstate: current_term: 1 leader_uuid: "e90437df78a34b5a854338321873a93f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } health_report { overall_health: HEALTHY } } }
I20260809 09:28:05.829015 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.826046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.9.102.3:35891)
0809 09:28:05.826194 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.826198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.826213 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:28:05.826261 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.826264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.826311 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.826390 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:28:05.826636 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.827171 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.827968 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.828226 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.828338 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.828598 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.828662 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.828757 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.828843 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.828903 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 09:28:05.830699 22642 raft_consensus.cc:1261] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Refusing update from remote peer e90437df78a34b5a854338321873a93f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:05.830724 22881 raft_consensus.cc:1261] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Refusing update from remote peer e90437df78a34b5a854338321873a93f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:05.831058 23242 consensus_queue.cc:1035] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:05.831261 23245 consensus_queue.cc:1035] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:05.838904 22352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:28:05.839089 22352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:28:05.840920 22352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f: Sending DeleteTablet for 3 replicas of tablet 2d8d14023a5e43229cc115f32a4cd212
I20260809 09:28:05.841387 22597 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8d14023a5e43229cc115f32a4cd212 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:05 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:54802
I20260809 09:28:05.841455 22838 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8d14023a5e43229cc115f32a4cd212 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:05 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:50102
I20260809 09:28:05.841568 23268 tablet_replica.cc:330] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584: stopping tablet replica
I20260809 09:28:05.841683 23268 raft_consensus.cc:2229] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:28:05.841696 23085 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d8d14023a5e43229cc115f32a4cd212 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:05 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:36642
I20260809 09:28:05.841835 23268 raft_consensus.cc:2258] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:05.841877 23269 tablet_replica.cc:330] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f: stopping tablet replica
I20260809 09:28:05.841985 23266 tablet_replica.cc:330] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79: stopping tablet replica
I20260809 09:28:05.842023 23269 raft_consensus.cc:2229] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:28:05.842129 23266 raft_consensus.cc:2229] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:28:05.842201 23269 raft_consensus.cc:2258] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:05.842269 23266 raft_consensus.cc:2258] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:05.842826 23269 ts_tablet_manager.cc:1905] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:05.843097 23268 ts_tablet_manager.cc:1905] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:05.844906 23268 ts_tablet_manager.cc:1918] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:28:05.844997 23268 log.cc:1196] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-0/wal/wals/2d8d14023a5e43229cc115f32a4cd212
I20260809 09:28:05.845094 23269 ts_tablet_manager.cc:1918] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:28:05.845172 23269 log.cc:1196] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/2d8d14023a5e43229cc115f32a4cd212
I20260809 09:28:05.845281 23268 ts_tablet_manager.cc:1939] T 2d8d14023a5e43229cc115f32a4cd212 P 2cfe2644bebf4251930b0be0d02c2584: Deleting consensus metadata
I20260809 09:28:05.845418 23269 ts_tablet_manager.cc:1939] T 2d8d14023a5e43229cc115f32a4cd212 P e90437df78a34b5a854338321873a93f: Deleting consensus metadata
I20260809 09:28:05.845955 22333 catalog_manager.cc:4841] TS 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443): tablet 2d8d14023a5e43229cc115f32a4cd212 (table test-table [id=ca81a786a56c4a53b765bafb1b2cdd05]) successfully deleted
I20260809 09:28:05.846042 22338 catalog_manager.cc:4841] TS e90437df78a34b5a854338321873a93f (127.9.102.3:35891): tablet 2d8d14023a5e43229cc115f32a4cd212 (table test-table [id=ca81a786a56c4a53b765bafb1b2cdd05]) successfully deleted
I20260809 09:28:05.846024 22352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 09:28:05.846405 23266 ts_tablet_manager.cc:1905] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 09:28:05.846396 22352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:28:05.848793 23266 ts_tablet_manager.cc:1918] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:28:05.848875 23266 log.cc:1196] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/2d8d14023a5e43229cc115f32a4cd212
I20260809 09:28:05.849125 23266 ts_tablet_manager.cc:1939] T 2d8d14023a5e43229cc115f32a4cd212 P 4869fad624c04f58927b0d002df8ce79: Deleting consensus metadata
I20260809 09:28:05.849949 22338 catalog_manager.cc:4841] TS 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759): tablet 2d8d14023a5e43229cc115f32a4cd212 (table test-table [id=ca81a786a56c4a53b765bafb1b2cdd05]) successfully deleted
I20260809 09:28:05.850903 22597 tablet_service.cc:1467] Processing CreateTablet for tablet bc3595f6fcc74620ab57a11f147c6af6 (DEFAULT_TABLE table=test-table [id=b257697f42294c2e95f153f5225952dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:05.851150 22597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3595f6fcc74620ab57a11f147c6af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.851123 23085 tablet_service.cc:1467] Processing CreateTablet for tablet bc3595f6fcc74620ab57a11f147c6af6 (DEFAULT_TABLE table=test-table [id=b257697f42294c2e95f153f5225952dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:05.851155 22838 tablet_service.cc:1467] Processing CreateTablet for tablet bc3595f6fcc74620ab57a11f147c6af6 (DEFAULT_TABLE table=test-table [id=b257697f42294c2e95f153f5225952dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:05.851310 23085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3595f6fcc74620ab57a11f147c6af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.851341 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3595f6fcc74620ab57a11f147c6af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:05.853431 23233 tablet_bootstrap.cc:492] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584: Bootstrap starting.
I20260809 09:28:05.854416 23233 tablet_bootstrap.cc:654] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.855433 23232 tablet_bootstrap.cc:492] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f: Bootstrap starting.
I20260809 09:28:05.855512 23233 tablet_bootstrap.cc:492] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584: No bootstrap required, opened a new log
I20260809 09:28:05.855577 23233 ts_tablet_manager.cc:1397] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260809 09:28:05.855926 23233 raft_consensus.cc:348] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.856021 23233 raft_consensus.cc:374] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.856048 23233 raft_consensus.cc:729] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cfe2644bebf4251930b0be0d02c2584, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.856184 23233 consensus_queue.cc:260] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.856231 23232 tablet_bootstrap.cc:654] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.856415 23233 ts_tablet_manager.cc:1428] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:28:05.857081 23235 tablet_bootstrap.cc:492] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79: Bootstrap starting.
I20260809 09:28:05.857237 23232 tablet_bootstrap.cc:492] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f: No bootstrap required, opened a new log
I20260809 09:28:05.857300 23232 ts_tablet_manager.cc:1397] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260809 09:28:05.857609 23232 raft_consensus.cc:348] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.857692 23232 raft_consensus.cc:374] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.857717 23232 raft_consensus.cc:729] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90437df78a34b5a854338321873a93f, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.857822 23235 tablet_bootstrap.cc:654] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:05.857946 23232 consensus_queue.cc:260] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.858845 23232 ts_tablet_manager.cc:1428] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260809 09:28:05.858969 23235 tablet_bootstrap.cc:492] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79: No bootstrap required, opened a new log
I20260809 09:28:05.859046 23235 ts_tablet_manager.cc:1397] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:28:05.859494 23235 raft_consensus.cc:348] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.859649 23235 raft_consensus.cc:374] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:05.859700 23235 raft_consensus.cc:729] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4869fad624c04f58927b0d002df8ce79, State: Initialized, Role: FOLLOWER
I20260809 09:28:05.859831 23235 consensus_queue.cc:260] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.861208 23235 ts_tablet_manager.cc:1428] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260809 09:28:05.911407 23245 raft_consensus.cc:482] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:28:05.911507 23245 raft_consensus.cc:504] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.911826 23245 leader_election.cc:288] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443), 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759)
I20260809 09:28:05.912189 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3595f6fcc74620ab57a11f147c6af6" candidate_uuid: "e90437df78a34b5a854338321873a93f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cfe2644bebf4251930b0be0d02c2584" is_pre_election: true
I20260809 09:28:05.912233 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3595f6fcc74620ab57a11f147c6af6" candidate_uuid: "e90437df78a34b5a854338321873a93f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4869fad624c04f58927b0d002df8ce79" is_pre_election: true
I20260809 09:28:05.912344 22642 raft_consensus.cc:2454] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e90437df78a34b5a854338321873a93f in term 0.
I20260809 09:28:05.912357 22881 raft_consensus.cc:2454] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e90437df78a34b5a854338321873a93f in term 0.
I20260809 09:28:05.912643 22997 leader_election.cc:302] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cfe2644bebf4251930b0be0d02c2584, e90437df78a34b5a854338321873a93f; no voters:
I20260809 09:28:05.912827 23245 raft_consensus.cc:2781] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:28:05.912885 23245 raft_consensus.cc:482] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:28:05.912933 23245 raft_consensus.cc:3037] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.913830 23245 raft_consensus.cc:504] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.914171 23245 leader_election.cc:288] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [CANDIDATE]: Term 1 election: Requested vote from peers 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443), 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759)
I20260809 09:28:05.914429 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3595f6fcc74620ab57a11f147c6af6" candidate_uuid: "e90437df78a34b5a854338321873a93f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cfe2644bebf4251930b0be0d02c2584"
I20260809 09:28:05.914544 22642 raft_consensus.cc:3037] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.914525 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc3595f6fcc74620ab57a11f147c6af6" candidate_uuid: "e90437df78a34b5a854338321873a93f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4869fad624c04f58927b0d002df8ce79"
I20260809 09:28:05.914651 22881 raft_consensus.cc:3037] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:05.915350 22642 raft_consensus.cc:2454] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e90437df78a34b5a854338321873a93f in term 1.
I20260809 09:28:05.915438 22881 raft_consensus.cc:2454] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e90437df78a34b5a854338321873a93f in term 1.
I20260809 09:28:05.915606 22997 leader_election.cc:302] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cfe2644bebf4251930b0be0d02c2584, e90437df78a34b5a854338321873a93f; no voters:
I20260809 09:28:05.915777 23245 raft_consensus.cc:2781] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:28:05.915860 23245 raft_consensus.cc:686] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 1 LEADER]: Becoming Leader. State: Replica: e90437df78a34b5a854338321873a93f, State: Running, Role: LEADER
I20260809 09:28:05.915966 23245 consensus_queue.cc:237] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } }
I20260809 09:28:05.917248 22350 catalog_manager.cc:5495] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f reported cstate change: term changed from 0 to 1, leader changed from <none> to e90437df78a34b5a854338321873a93f (127.9.102.3). New cstate: current_term: 1 leader_uuid: "e90437df78a34b5a854338321873a93f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e90437df78a34b5a854338321873a93f" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 35891 } health_report { overall_health: HEALTHY } } }
I20260809 09:28:05.951885 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.949321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:33409 (local address 127.9.102.2:35759)
0809 09:28:05.949410 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.949414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.949576 (+ 162us) server_negotiation.cc:407] Connection header received
0809 09:28:05.949672 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.949673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.949709 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.949767 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:28:05.949939 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.950368 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.951083 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.951262 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.951329 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.951564 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.951617 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.951666 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.951751 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.951788 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260809 09:28:05.951895 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.949243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:35759 (local address 127.9.102.3:33409)
0809 09:28:05.949506 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.949517 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.949594 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.949791 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.949793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.949798 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 09:28:05.949878 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.949882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.950444 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.950446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.950980 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.950996 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.951351 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.951353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.951410 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.951447 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.951696 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.951700 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.951743 (+ 43us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.951801 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260809 09:28:05.956655 22642 raft_consensus.cc:1261] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Refusing update from remote peer e90437df78a34b5a854338321873a93f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:05.956670 22881 raft_consensus.cc:1261] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Refusing update from remote peer e90437df78a34b5a854338321873a93f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:05.956961 23258 consensus_queue.cc:1035] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cfe2644bebf4251930b0be0d02c2584" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:05.957093 23245 consensus_queue.cc:1035] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4869fad624c04f58927b0d002df8ce79" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:05.972234 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.969503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:53479 (local address 127.9.102.3:35891)
0809 09:28:05.969584 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:05.969588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:05.969600 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:28:05.969653 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:05.969654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:05.969686 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:05.969769 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:28:05.969972 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.970386 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.971172 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.971453 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.971558 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.971808 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.971878 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.971963 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:05.972046 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 09:28:05.972122 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260809 09:28:05.972241 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:05.969396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:35891 (local address 127.9.102.2:53479)
0809 09:28:05.969494 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:05.969508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:05.969579 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:05.969779 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:05.969782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:05.969788 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 09:28:05.969866 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.969871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.970501 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.970505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.971021 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:05.971029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:05.971569 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:05.971572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:05.971649 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:05.971690 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 09:28:05.971968 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 09:28:05.971976 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 09:28:05.972036 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 09:28:05.972146 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260809 09:28:05.986704 22351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:28:05.986804 22351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:28:05.988436 22351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bedbddb15d434b389fe0388919dabd8f: Sending DeleteTablet for 3 replicas of tablet bc3595f6fcc74620ab57a11f147c6af6
I20260809 09:28:05.988873 22597 tablet_service.cc:1514] Processing DeleteTablet for tablet bc3595f6fcc74620ab57a11f147c6af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:05 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:54802
I20260809 09:28:05.989048 23268 tablet_replica.cc:330] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584: stopping tablet replica
I20260809 09:28:05.989117 23085 tablet_service.cc:1514] Processing DeleteTablet for tablet bc3595f6fcc74620ab57a11f147c6af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:05 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:36642
I20260809 09:28:05.989171 23268 raft_consensus.cc:2229] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:28:05.989282 23268 raft_consensus.cc:2258] T bc3595f6fcc74620ab57a11f147c6af6 P 2cfe2644bebf4251930b0be0d02c2584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:05.989300 23269 tablet_replica.cc:330] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f: stopping tablet replica
I20260809 09:28:05.989408 23269 raft_consensus.cc:2229] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:28:05.989470 22838 tablet_service.cc:1514] Processing DeleteTablet for tablet bc3595f6fcc74620ab57a11f147c6af6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:05 UTC) from {username='oryx', principal='oryx/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:50102
I20260809 09:28:05.989581 23269 raft_consensus.cc:2258] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:05.989629 23266 tablet_replica.cc:330] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79: stopping tablet replica
I20260809 09:28:05.989711 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 22489
I20260809 09:28:05.989720 23266 raft_consensus.cc:2229] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:28:05.989861 23266 raft_consensus.cc:2258] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:05.995301 23266 ts_tablet_manager.cc:1905] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:05.995774 23269 ts_tablet_manager.cc:1905] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:05.997397 23266 ts_tablet_manager.cc:1918] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:28:05.997519 23266 log.cc:1196] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/bc3595f6fcc74620ab57a11f147c6af6
I20260809 09:28:05.997738 23266 ts_tablet_manager.cc:1939] T bc3595f6fcc74620ab57a11f147c6af6 P 4869fad624c04f58927b0d002df8ce79: Deleting consensus metadata
I20260809 09:28:05.998373 22338 catalog_manager.cc:4841] TS 4869fad624c04f58927b0d002df8ce79 (127.9.102.2:35759): tablet bc3595f6fcc74620ab57a11f147c6af6 (table test-table [id=b257697f42294c2e95f153f5225952dc]) successfully deleted
I20260809 09:28:05.999004 23269 ts_tablet_manager.cc:1918] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 09:28:05.999123 23269 log.cc:1196] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/bc3595f6fcc74620ab57a11f147c6af6
I20260809 09:28:05.999406 23269 ts_tablet_manager.cc:1939] T bc3595f6fcc74620ab57a11f147c6af6 P e90437df78a34b5a854338321873a93f: Deleting consensus metadata
I20260809 09:28:06.000128 22338 catalog_manager.cc:4841] TS e90437df78a34b5a854338321873a93f (127.9.102.3:35891): tablet bc3595f6fcc74620ab57a11f147c6af6 (table test-table [id=b257697f42294c2e95f153f5225952dc]) successfully deleted
W20260809 09:28:06.006636 22333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:38443 (error 108)
I20260809 09:28:06.006912 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 22737
I20260809 09:28:06.013140 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:06.012389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38443 (local address 127.0.0.1:54822)
0809 09:28:06.012973 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:06.013045 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:38443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":408,"threads_started":1}
W20260809 09:28:06.013345 22333 catalog_manager.cc:4584] TS 2cfe2644bebf4251930b0be0d02c2584 (127.9.102.1:38443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc3595f6fcc74620ab57a11f147c6af6: Network error: Client connection negotiation failed: client connection to 127.9.102.1:38443: connect: Connection refused (error 111)
I20260809 09:28:06.018648 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 22975
I20260809 09:28:06.035202 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 22307
2026-08-09T09:28:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3078 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:28:06 dist-test-slave-7vjv krb5kdc[23383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:28:06 dist-test-slave-7vjv krb5kdc[23383](info): set up 2 sockets
Aug 09 09:28:06 dist-test-slave-7vjv krb5kdc[23383](info): commencing operation
krb5kdc: starting...
W20260809 09:28:07.195685 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:28:07 dist-test-slave-7vjv krb5kdc[23383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:28:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:28:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:28:07.308648 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:37463
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:07.407663 23401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:07.407951 23401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:07.408027 23401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:07.411448 23401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:28:07.411538 23401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:07.411571 23401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:07.411653 23401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:28:07.411686 23401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:28:07.416303 23401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:37463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:07.417140 23401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:07.418079 23401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:07.426131 23414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:07.426115 23411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:07.426261 23401 server_base.cc:1034] running on GCE node
W20260809 09:28:07.426424 23416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:07.426925 23401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:07.427842 23401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:07.429011 23401 hybrid_clock.cc:648] HybridClock initialized: now 1786267687428985 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T09:28:07Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1397 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:28:07 dist-test-slave-7vjv krb5kdc[23422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:28:07 dist-test-slave-7vjv krb5kdc[23422](info): set up 2 sockets
Aug 09 09:28:07 dist-test-slave-7vjv krb5kdc[23422](info): commencing operation
krb5kdc: starting...
W20260809 09:28:08.580554 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:28:08 dist-test-slave-7vjv krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:28:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:28:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:28:08.695623 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:33163
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:33163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:08.797686 23448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:08.797959 23448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:08.798026 23448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:08.801668 23448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:28:08.801749 23448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:08.801783 23448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:08.801837 23448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:28:08.801869 23448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:28:08.806277 23448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:33163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:08.807132 23448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:08.807971 23448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:08.815299 23460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:08.815436 23467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:08.815467 23448 server_base.cc:1034] running on GCE node
W20260809 09:28:08.815299 23464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:08.816246 23448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:08.817173 23448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:08.818352 23448 hybrid_clock.cc:648] HybridClock initialized: now 1786267688818324 us; error 38 us; skew 500 ppm
Aug 09 09:28:08 dist-test-slave-7vjv krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267688, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:08.821101 23448 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:28:08.821821 23448 webserver.cc:460] Webserver started at http://127.9.102.62:38347/ using document root <none> and password file <none>
I20260809 09:28:08.822293 23448 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:08.822399 23448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:08.822638 23448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:08.824484 23448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "015db9c1b58b44bba0ae0b0cab354aee"
format_stamp: "Formatted at 2026-08-09 09:28:08 on dist-test-slave-7vjv"
server_key: "558c55ba4d5c99c70badad5b45fd1e5b"
server_key_iv: "d7201ffb147f6ae299da4d675b9da850"
server_key_version: "encryptionkey@0"
I20260809 09:28:08.825003 23448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "015db9c1b58b44bba0ae0b0cab354aee"
format_stamp: "Formatted at 2026-08-09 09:28:08 on dist-test-slave-7vjv"
server_key: "558c55ba4d5c99c70badad5b45fd1e5b"
server_key_iv: "d7201ffb147f6ae299da4d675b9da850"
server_key_version: "encryptionkey@0"
I20260809 09:28:08.828362 23448 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:28:08.830331 23475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:08.831316 23448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:28:08.831486 23448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "015db9c1b58b44bba0ae0b0cab354aee"
format_stamp: "Formatted at 2026-08-09 09:28:08 on dist-test-slave-7vjv"
server_key: "558c55ba4d5c99c70badad5b45fd1e5b"
server_key_iv: "d7201ffb147f6ae299da4d675b9da850"
server_key_version: "encryptionkey@0"
I20260809 09:28:08.831593 23448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:08.842006 23448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:08.844560 23448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:08.844748 23448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:08.850898 23448 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:33163
I20260809 09:28:08.850924 23561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:33163 every 8 connection(s)
I20260809 09:28:08.851694 23448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:28:08.854099 23562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:08.858711 23562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee: Bootstrap starting.
I20260809 09:28:08.860890 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 23448
I20260809 09:28:08.861022 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:28:08.861133 23562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:08.861330 9624 external_mini_cluster.cc:1428] Setting key 7fa67f906776b3ed218787716fd73471
I20260809 09:28:08.861939 23562 log.cc:824] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee: Log is configured to *not* fsync() on all Append() calls
I20260809 09:28:08.863977 23562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee: No bootstrap required, opened a new log
I20260809 09:28:08.867794 23562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015db9c1b58b44bba0ae0b0cab354aee" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33163 } }
I20260809 09:28:08.867996 23562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:08.868083 23562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015db9c1b58b44bba0ae0b0cab354aee, State: Initialized, Role: FOLLOWER
I20260809 09:28:08.868517 23562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015db9c1b58b44bba0ae0b0cab354aee" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33163 } }
I20260809 09:28:08.868696 23562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:28:08.868765 23562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:28:08.868847 23562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 0 FOLLOWER]: Advancing to term 1
Aug 09 09:28:08 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:28:08.869626 23562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015db9c1b58b44bba0ae0b0cab354aee" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33163 } }
I20260809 09:28:08.869915 23562 leader_election.cc:302] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 015db9c1b58b44bba0ae0b0cab354aee; no voters:
I20260809 09:28:08.870223 23562 leader_election.cc:288] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:28:08.870317 23567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:28:08.870551 23567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 1 LEADER]: Becoming Leader. State: Replica: 015db9c1b58b44bba0ae0b0cab354aee, State: Running, Role: LEADER
I20260809 09:28:08.870824 23567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015db9c1b58b44bba0ae0b0cab354aee" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33163 } }
I20260809 09:28:08.871446 23562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:28:08.871901 23577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 015db9c1b58b44bba0ae0b0cab354aee. Latest consensus state: current_term: 1 leader_uuid: "015db9c1b58b44bba0ae0b0cab354aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015db9c1b58b44bba0ae0b0cab354aee" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33163 } } }
I20260809 09:28:08.872013 23577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:08.872509 23571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "015db9c1b58b44bba0ae0b0cab354aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015db9c1b58b44bba0ae0b0cab354aee" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33163 } } }
I20260809 09:28:08.872665 23571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:08.875082 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:08.862328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.9.102.62:33163)
0809 09:28:08.862673 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:08.862691 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:08.862731 (+ 40us) server_negotiation.cc:407] Connection header received
0809 09:28:08.863455 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:08.863475 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:08.863806 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:08.864171 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:08.864771 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:08.866069 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:08.866834 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:08.867458 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:08.867605 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:08.870051 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:08.870057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:08.870069 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:08.870095 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:08.871932 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:08.872423 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:08.872428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:08.872432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:08.872554 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:08.872855 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:08.872858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:08.872859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:08.873287 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:08.873464 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:08.874041 (+ 577us) server_negotiation.cc:299] Negotiation successful
0809 09:28:08.874318 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":121,"threads_started":1}
W20260809 09:28:08.876444 23602 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:28:08.876542 23602 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:28:08.876606 23581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:28:08.877424 23581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:28:08.882498 23581 catalog_manager.cc:1347] Generated new cluster ID: 8c0d15cbd59240a9b88ac22fa3d6710f
I20260809 09:28:08.882566 23581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:28:08.897181 23581 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:28:08.897958 23581 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:28:08.902108 23581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee: Generated new TSK 0
I20260809 09:28:08.902657 23581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:28:08.931250 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:33163
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:09.031519 23610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:09.031757 23610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:09.031826 23610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:09.035709 23610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:09.035785 23610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:09.035866 23610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:28:09.040474 23610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:09.041383 23610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:09.042239 23610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:09.050745 23623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:09.050799 23617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:09.050742 23619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:09.050997 23610 server_base.cc:1034] running on GCE node
I20260809 09:28:09.051393 23610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:09.051980 23610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:09.053176 23610 hybrid_clock.cc:648] HybridClock initialized: now 1786267689053163 us; error 33 us; skew 500 ppm
Aug 09 09:28:09 dist-test-slave-7vjv krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267689, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:09.055619 23610 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:28:09.056511 23610 webserver.cc:460] Webserver started at http://127.9.102.1:35041/ using document root <none> and password file <none>
I20260809 09:28:09.057042 23610 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:09.057142 23610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:09.057374 23610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:09.059089 23610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "fdf5e987c0de4491bb05e15ae455ee07"
format_stamp: "Formatted at 2026-08-09 09:28:09 on dist-test-slave-7vjv"
server_key: "18ee78f4fb4021692ab66cfbf2301fba"
server_key_iv: "e77239bb74824889c68cc744eefde5a8"
server_key_version: "encryptionkey@0"
I20260809 09:28:09.059652 23610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "fdf5e987c0de4491bb05e15ae455ee07"
format_stamp: "Formatted at 2026-08-09 09:28:09 on dist-test-slave-7vjv"
server_key: "18ee78f4fb4021692ab66cfbf2301fba"
server_key_iv: "e77239bb74824889c68cc744eefde5a8"
server_key_version: "encryptionkey@0"
I20260809 09:28:09.063007 23610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 09:28:09.065207 23631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.066262 23610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:28:09.066406 23610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "fdf5e987c0de4491bb05e15ae455ee07"
format_stamp: "Formatted at 2026-08-09 09:28:09 on dist-test-slave-7vjv"
server_key: "18ee78f4fb4021692ab66cfbf2301fba"
server_key_iv: "e77239bb74824889c68cc744eefde5a8"
server_key_version: "encryptionkey@0"
I20260809 09:28:09.066511 23610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:09.081979 23610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:09.084821 23610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:09.084949 23610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:09.085366 23610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:09.085994 23610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:09.086045 23610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.086077 23610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:09.086091 23610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.094017 23610 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:45047
I20260809 09:28:09.094043 23833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:45047 every 8 connection(s)
I20260809 09:28:09.094770 23610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:28:09.096737 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 23610
I20260809 09:28:09.096868 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:28:09.097203 9624 external_mini_cluster.cc:1428] Setting key 32c452ded16a0b43009c46d1d81a3590
Aug 09 09:28:09 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267689, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:28:09.107949 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:09.096428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:49869 (local address 127.9.102.62:33163)
0809 09:28:09.096540 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:09.096544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:09.097584 (+ 1040us) server_negotiation.cc:407] Connection header received
0809 09:28:09.098439 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:09.098442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:09.098490 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:09.098572 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:09.099978 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.100564 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.101631 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.101849 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.101964 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:09.104569 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:09.104572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:09.104574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:09.104595 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:09.105937 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:09.106461 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:09.106464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:09.106466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:09.106550 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:09.106905 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:09.106909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:09.106911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:09.107131 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:09.107238 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:09.107682 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 09:28:09.107793 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 09:28:09.108711 23838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:09.096728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:49869)
0809 09:28:09.097408 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:09.097455 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:09.098202 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:09.098744 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:09.098751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:09.099195 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:09.099826 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:09.099847 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.100715 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.100719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:09.101492 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:09.101501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.102000 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.102004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:09.102163 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:09.102676 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:09.102690 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:09.104410 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:09.106094 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:09.106099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:09.106111 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:09.106354 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:09.106673 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:09.106676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:09.106678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:09.106807 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:09.107239 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:09.107246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:09.107457 (+ 211us) client_negotiation.cc:769] Sending connection context
0809 09:28:09.107638 (+ 181us) client_negotiation.cc:240] Negotiation successful
0809 09:28:09.107904 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":142,"threads_started":1}
I20260809 09:28:09.109865 23834 heartbeater.cc:347] Connected to a master server at 127.9.102.62:33163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
I20260809 09:28:09.110100 23834 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:09.110649 23834 heartbeater.cc:511] Master 127.9.102.62:33163 requested a full tablet report, sending...
I20260809 09:28:09.112215 23504 ts_manager.cc:194] Registered new tserver with Master: fdf5e987c0de4491bb05e15ae455ee07 (127.9.102.1:45047)
I20260809 09:28:09.113740 23504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:49869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:28:09.125293 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:33163
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:09.222115 23843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:09.222352 23843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:09.222424 23843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:09.225875 23843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:09.225956 23843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:09.226034 23843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:28:09.230401 23843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:09.231244 23843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:09.232091 23843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:09.240763 23855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:09.240900 23843 server_base.cc:1034] running on GCE node
W20260809 09:28:09.240773 23853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:09.240808 23852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:09.241377 23843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:09.241999 23843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:09.243182 23843 hybrid_clock.cc:648] HybridClock initialized: now 1786267689243166 us; error 27 us; skew 500 ppm
Aug 09 09:28:09 dist-test-slave-7vjv krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267689, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:09.245721 23843 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:28:09.246465 23843 webserver.cc:460] Webserver started at http://127.9.102.2:41673/ using document root <none> and password file <none>
I20260809 09:28:09.246981 23843 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:09.247082 23843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:09.247304 23843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:09.249126 23843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "bbb489f8a90c41b5a1cc9fb81fdbab6a"
format_stamp: "Formatted at 2026-08-09 09:28:09 on dist-test-slave-7vjv"
server_key: "37bf36648a1f299f24bd4ebff51c8b81"
server_key_iv: "2ba0c7137797e8e04adaacb3cb9f0011"
server_key_version: "encryptionkey@0"
I20260809 09:28:09.249647 23843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "bbb489f8a90c41b5a1cc9fb81fdbab6a"
format_stamp: "Formatted at 2026-08-09 09:28:09 on dist-test-slave-7vjv"
server_key: "37bf36648a1f299f24bd4ebff51c8b81"
server_key_iv: "2ba0c7137797e8e04adaacb3cb9f0011"
server_key_version: "encryptionkey@0"
I20260809 09:28:09.252965 23843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 09:28:09.255188 23869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.256304 23843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:28:09.256436 23843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "bbb489f8a90c41b5a1cc9fb81fdbab6a"
format_stamp: "Formatted at 2026-08-09 09:28:09 on dist-test-slave-7vjv"
server_key: "37bf36648a1f299f24bd4ebff51c8b81"
server_key_iv: "2ba0c7137797e8e04adaacb3cb9f0011"
server_key_version: "encryptionkey@0"
I20260809 09:28:09.256563 23843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:09.277493 23843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:09.280654 23843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:09.280860 23843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:09.281416 23843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:09.282166 23843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:09.282244 23843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.282308 23843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:09.282361 23843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.290938 23843 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:37967
I20260809 09:28:09.290962 24039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:37967 every 8 connection(s)
I20260809 09:28:09.291689 23843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
Aug 09 09:28:09 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267689, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:28:09.300886 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 23843
I20260809 09:28:09.301067 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:28:09.301421 9624 external_mini_cluster.cc:1428] Setting key 1d951c4ea03503b50e976495df36a1ab
I20260809 09:28:09.304292 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:09.293311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:51429 (local address 127.9.102.62:33163)
0809 09:28:09.293447 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:09.293450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:09.294412 (+ 962us) server_negotiation.cc:407] Connection header received
0809 09:28:09.295165 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:09.295168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:09.295213 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:09.295283 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:09.296331 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.296801 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.297785 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.297978 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.298056 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:09.300244 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:09.300246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:09.300248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:09.300271 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:09.301901 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:09.302617 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:09.302623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:09.302627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:09.302724 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:09.303095 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:09.303098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:09.303099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:09.303338 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:09.303447 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:09.303875 (+ 428us) server_negotiation.cc:299] Negotiation successful
0809 09:28:09.304013 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 09:28:09.304831 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:09.293605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:51429)
0809 09:28:09.294259 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:09.294295 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:09.294987 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:09.295438 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:09.295444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:09.295750 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:09.296203 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:09.296215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.296911 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.296913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:09.297677 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:09.297686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.298103 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.298105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:09.298223 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:09.298653 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:09.298667 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:09.300031 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:09.302186 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:09.302191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:09.302203 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:09.302497 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:09.302832 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:09.302835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:09.302836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:09.302973 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:09.303447 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:09.303452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:09.303646 (+ 194us) client_negotiation.cc:769] Sending connection context
0809 09:28:09.303785 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 09:28:09.303993 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":117,"threads_started":1}
I20260809 09:28:09.306020 24040 heartbeater.cc:347] Connected to a master server at 127.9.102.62:33163
I20260809 09:28:09.306236 24040 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:09.306639 24040 heartbeater.cc:511] Master 127.9.102.62:33163 requested a full tablet report, sending...
I20260809 09:28:09.307663 23504 ts_manager.cc:194] Registered new tserver with Master: bbb489f8a90c41b5a1cc9fb81fdbab6a (127.9.102.2:37967)
I20260809 09:28:09.308297 23504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:51429
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:28:09.329144 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:33163
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:09.429591 24049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:09.429778 24049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:09.429831 24049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:09.433395 24049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:09.433467 24049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:09.433552 24049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:28:09.438263 24049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:09.439024 24049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:09.439826 24049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 09:28:09.448395 24049 server_base.cc:1034] running on GCE node
W20260809 09:28:09.448344 24058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:09.448321 24056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:09.448307 24055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:09.448868 24049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:09.449426 24049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:09.450590 24049 hybrid_clock.cc:648] HybridClock initialized: now 1786267689450573 us; error 28 us; skew 500 ppm
Aug 09 09:28:09 dist-test-slave-7vjv krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267689, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:09.453250 24049 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:28:09.453970 24049 webserver.cc:460] Webserver started at http://127.9.102.3:46705/ using document root <none> and password file <none>
I20260809 09:28:09.454459 24049 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:09.454515 24049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:09.454722 24049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:09.456609 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "743c2e89af924a28a4b2c3668f304b3e"
format_stamp: "Formatted at 2026-08-09 09:28:09 on dist-test-slave-7vjv"
server_key: "35958bdb62fdf1e0a98869864e10606e"
server_key_iv: "c1e42f7340edda769b6e1d0dfdad71ff"
server_key_version: "encryptionkey@0"
I20260809 09:28:09.457119 24049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "743c2e89af924a28a4b2c3668f304b3e"
format_stamp: "Formatted at 2026-08-09 09:28:09 on dist-test-slave-7vjv"
server_key: "35958bdb62fdf1e0a98869864e10606e"
server_key_iv: "c1e42f7340edda769b6e1d0dfdad71ff"
server_key_version: "encryptionkey@0"
I20260809 09:28:09.460556 24049 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:28:09.462693 24070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.463732 24049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:28:09.463864 24049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "743c2e89af924a28a4b2c3668f304b3e"
format_stamp: "Formatted at 2026-08-09 09:28:09 on dist-test-slave-7vjv"
server_key: "35958bdb62fdf1e0a98869864e10606e"
server_key_iv: "c1e42f7340edda769b6e1d0dfdad71ff"
server_key_version: "encryptionkey@0"
I20260809 09:28:09.463967 24049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:09.481891 24049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:09.484390 24049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:09.484575 24049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:09.485096 24049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:09.485834 24049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:09.485913 24049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.485977 24049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:09.486033 24049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.494642 24049 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:38719
I20260809 09:28:09.494665 24262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:38719 every 8 connection(s)
I20260809 09:28:09.495395 24049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
Aug 09 09:28:09 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267689, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:28:09.504558 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 24049
I20260809 09:28:09.504696 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:28:09.504956 9624 external_mini_cluster.cc:1428] Setting key 1fbfa1f148d7dbca83a243ac643a4a44
I20260809 09:28:09.507490 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:09.497051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:44539 (local address 127.9.102.62:33163)
0809 09:28:09.497217 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:09.497222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:09.498086 (+ 864us) server_negotiation.cc:407] Connection header received
0809 09:28:09.498789 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:09.498792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:09.498834 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:09.498902 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:09.499945 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.500547 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.501520 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.501700 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.501766 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:09.503920 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:09.503922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:09.503924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:09.503945 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:09.505386 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:09.505880 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:09.505882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:09.505884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:09.505965 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:09.506382 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:09.506385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:09.506387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:09.506564 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:09.506675 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:09.507219 (+ 544us) server_negotiation.cc:299] Negotiation successful
0809 09:28:09.507330 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 09:28:09.508256 24266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:09.497367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:44539)
0809 09:28:09.497944 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:09.497979 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:09.498630 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:09.499053 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:09.499058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:09.499347 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:09.499819 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:09.499831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.500653 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.500656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:09.501418 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:09.501426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.501813 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.501815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:09.501930 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:09.502438 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:09.502452 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:09.503775 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:09.505498 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:09.505502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:09.505511 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:09.505784 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:09.506096 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:09.506098 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:09.506100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:09.506279 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:09.506673 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:09.506679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:09.506954 (+ 275us) client_negotiation.cc:769] Sending connection context
0809 09:28:09.507176 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 09:28:09.507450 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":110,"threads_started":1}
I20260809 09:28:09.509238 24263 heartbeater.cc:347] Connected to a master server at 127.9.102.62:33163
I20260809 09:28:09.509462 24263 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:09.509889 24263 heartbeater.cc:511] Master 127.9.102.62:33163 requested a full tablet report, sending...
I20260809 09:28:09.510805 23504 ts_manager.cc:194] Registered new tserver with Master: 743c2e89af924a28a4b2c3668f304b3e (127.9.102.3:38719)
I20260809 09:28:09.511308 23504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:44539
I20260809 09:28:09.518452 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab.
Entry for principal oryx/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab.
WARNING: no policy specified for oryx/127.9.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab.
Entry for principal oryx/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab.
WARNING: no policy specified for oryx/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab.
Entry for principal oryx/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab.
I20260809 09:28:09.588937 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 23610
I20260809 09:28:09.603559 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 23843
I20260809 09:28:09.612427 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 24049
I20260809 09:28:09.625088 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 23448
I20260809 09:28:09.634657 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:33163
--webserver_interface=127.9.102.62
--webserver_port=38347
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:33163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:09.736629 24284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:09.736830 24284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:09.736917 24284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:09.740646 24284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:28:09.740700 24284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:09.740721 24284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:09.740741 24284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:28:09.740806 24284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:28:09.745482 24284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:33163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=38347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:09.746321 24284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:09.747215 24284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:09.754972 24293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:09.755055 24290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:09.755012 24291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:09.755231 24284 server_base.cc:1034] running on GCE node
I20260809 09:28:09.755841 24284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:09.756978 24284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:09.758210 24284 hybrid_clock.cc:648] HybridClock initialized: now 1786267689758178 us; error 44 us; skew 500 ppm
Aug 09 09:28:09 dist-test-slave-7vjv krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267689, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:09.760985 24284 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:28:09.761680 24284 webserver.cc:460] Webserver started at http://127.9.102.62:38347/ using document root <none> and password file <none>
I20260809 09:28:09.762104 24284 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:09.762187 24284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:09.765395 24284 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260809 09:28:09.767158 24304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.768342 24284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260809 09:28:09.768481 24284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "015db9c1b58b44bba0ae0b0cab354aee"
format_stamp: "Formatted at 2026-08-09 09:28:08 on dist-test-slave-7vjv"
server_key: "558c55ba4d5c99c70badad5b45fd1e5b"
server_key_iv: "d7201ffb147f6ae299da4d675b9da850"
server_key_version: "encryptionkey@0"
I20260809 09:28:09.768956 24284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:09.792284 24284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:09.795121 24284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:09.795320 24284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:09.801955 24284 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:33163
I20260809 09:28:09.801988 24382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:33163 every 8 connection(s)
I20260809 09:28:09.802851 24284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:28:09.806438 24383 sys_catalog.cc:263] Verifying existing consensus state
I20260809 09:28:09.807816 24383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee: Bootstrap starting.
I20260809 09:28:09.810783 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 24284
I20260809 09:28:09.811215 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:45047
--local_ip_for_outbound_sockets=127.9.102.1
--tserver_master_addrs=127.9.102.62:33163
--webserver_port=35041
--webserver_interface=127.9.102.1
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.102.1
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:28:09.826074 24383 log.cc:824] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee: Log is configured to *not* fsync() on all Append() calls
I20260809 09:28:09.833781 24383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 09:28:09.834200 24383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee: Bootstrap complete.
I20260809 09:28:09.839829 24383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015db9c1b58b44bba0ae0b0cab354aee" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33163 } }
I20260809 09:28:09.840518 24383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015db9c1b58b44bba0ae0b0cab354aee, State: Initialized, Role: FOLLOWER
I20260809 09:28:09.841061 24383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015db9c1b58b44bba0ae0b0cab354aee" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33163 } }
I20260809 09:28:09.841209 24383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:28:09.841267 24383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:28:09.841359 24383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 1 FOLLOWER]: Advancing to term 2
I20260809 09:28:09.843511 24383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015db9c1b58b44bba0ae0b0cab354aee" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33163 } }
I20260809 09:28:09.843839 24383 leader_election.cc:302] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 015db9c1b58b44bba0ae0b0cab354aee; no voters:
I20260809 09:28:09.844164 24383 leader_election.cc:288] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 09:28:09.844277 24387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 2 FOLLOWER]: Leader election won for term 2
I20260809 09:28:09.844532 24387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [term 2 LEADER]: Becoming Leader. State: Replica: 015db9c1b58b44bba0ae0b0cab354aee, State: Running, Role: LEADER
I20260809 09:28:09.844842 24387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015db9c1b58b44bba0ae0b0cab354aee" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33163 } }
I20260809 09:28:09.845275 24383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:28:09.845665 24389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 015db9c1b58b44bba0ae0b0cab354aee. Latest consensus state: current_term: 2 leader_uuid: "015db9c1b58b44bba0ae0b0cab354aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015db9c1b58b44bba0ae0b0cab354aee" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33163 } } }
I20260809 09:28:09.845862 24389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:09.845997 24388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "015db9c1b58b44bba0ae0b0cab354aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015db9c1b58b44bba0ae0b0cab354aee" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33163 } } }
I20260809 09:28:09.846084 24388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:09.846323 24397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:28:09.849195 24397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:28:09.850008 24397 catalog_manager.cc:1259] Loaded cluster ID: 8c0d15cbd59240a9b88ac22fa3d6710f
I20260809 09:28:09.850077 24397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:28:09.852353 24397 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:28:09.852959 24397 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 015db9c1b58b44bba0ae0b0cab354aee: Loaded TSK: 0
I20260809 09:28:09.854086 24397 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 09:28:09.924240 24385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:09.924472 24385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:09.924535 24385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:09.927822 24385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:09.927903 24385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:09.927975 24385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:28:09.932400 24385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.1.keytab
--principal=oryx/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:45047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=35041
--tserver_master_addrs=127.9.102.62:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:09.933296 24385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:09.934209 24385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:09.942634 24426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:09.942636 24425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:09.942606 24428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:09.943478 24385 server_base.cc:1034] running on GCE node
I20260809 09:28:09.943857 24385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:09.944530 24385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:09.945705 24385 hybrid_clock.cc:648] HybridClock initialized: now 1786267689945656 us; error 65 us; skew 500 ppm
Aug 09 09:28:09 dist-test-slave-7vjv krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267689, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:09.948698 24385 init.cc:377] Logged in from keytab as oryx/127.9.102.1@KRBTEST.COM (short username oryx)
I20260809 09:28:09.949473 24385 webserver.cc:460] Webserver started at http://127.9.102.1:35041/ using document root <none> and password file <none>
I20260809 09:28:09.949962 24385 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:09.950070 24385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:09.953804 24385 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:28:09.955781 24444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.956997 24385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:28:09.957177 24385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "fdf5e987c0de4491bb05e15ae455ee07"
format_stamp: "Formatted at 2026-08-09 09:28:09 on dist-test-slave-7vjv"
server_key: "18ee78f4fb4021692ab66cfbf2301fba"
server_key_iv: "e77239bb74824889c68cc744eefde5a8"
server_key_version: "encryptionkey@0"
I20260809 09:28:09.957741 24385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:09.977627 24385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:09.980499 24385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:09.980726 24385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:09.981320 24385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:09.982131 24385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:09.982221 24385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.982295 24385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:09.982332 24385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:09.990892 24385 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:45047
I20260809 09:28:09.990937 24626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:45047 every 8 connection(s)
I20260809 09:28:09.991869 24385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:28:09.998962 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 24385
I20260809 09:28:09.999459 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:37967
--local_ip_for_outbound_sockets=127.9.102.2
--tserver_master_addrs=127.9.102.62:33163
--webserver_port=41673
--webserver_interface=127.9.102.2
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.102.2
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 09:28:09 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:09 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:10.003814 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:09.993798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:48663)
0809 09:28:09.994379 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:09.994415 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:09.995122 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:09.996049 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:09.996055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:09.996415 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:09.996826 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:09.996850 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.997962 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.997964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:09.998914 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:09.998927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.999808 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.999812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.000003 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.000627 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:10.000643 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:10.003168 (+ 2525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":107,"threads_started":1}
W20260809 09:28:10.004292 24635 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:09.993739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:48663 (local address 127.9.102.62:33163)
0809 09:28:09.994303 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:09.994320 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:09.994533 (+ 213us) server_negotiation.cc:407] Connection header received
0809 09:28:09.995456 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:09.995466 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:09.995717 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:09.995951 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:09.997001 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.997847 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.999072 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:09.999503 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:09.999760 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.003441 (+ 3681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:48663: BlockingRecv error: recv got EOF from 127.9.102.1:48663 (error 108)
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":109,"threads_started":1}
W20260809 09:28:10.004405 24627 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:33163 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.102.62:33163: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:10.009879 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:10.004821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:39363)
0809 09:28:10.004903 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:10.004917 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:10.004979 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:10.005238 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:10.005240 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:10.005353 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:10.005428 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.005432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.006475 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.006477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.006935 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.006942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.007531 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.007534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.007595 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.008020 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:10.008029 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:10.009803 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260809 09:28:10.010119 24635 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:10.004904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:39363 (local address 127.9.102.62:33163)
0809 09:28:10.005018 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:10.005023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:10.005037 (+ 14us) server_negotiation.cc:407] Connection header received
0809 09:28:10.005076 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:10.005080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:10.005127 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:10.005215 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:10.005550 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.006366 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.007054 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.007419 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.007549 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.009984 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:39363: BlockingRecv error: recv got EOF from 127.9.102.1:39363 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:10.015003 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:10.010442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:46677)
0809 09:28:10.010520 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:10.010534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:10.010597 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:10.010891 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:10.010892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:10.010999 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:10.011064 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.011068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.011994 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.011996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.012466 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.012474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.013020 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.013022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.013078 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.013453 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:10.013463 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:10.014936 (+ 1473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260809 09:28:10.015162 24635 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:10.010536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:46677 (local address 127.9.102.62:33163)
0809 09:28:10.010662 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:10.010666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:10.010681 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:28:10.010718 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:10.010722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:10.010785 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:10.010874 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:10.011178 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.011893 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.012581 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.012918 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.013016 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.015073 (+ 2057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:46677: BlockingRecv error: recv got EOF from 127.9.102.1:46677 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 09:28:10.015306 24627 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 09:28:10.105271 24638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:10.105463 24638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:10.105496 24638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:10.108995 24638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:10.109048 24638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:10.109151 24638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:28:10.113631 24638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.2.keytab
--principal=oryx/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:37967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=41673
--tserver_master_addrs=127.9.102.62:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:10.114460 24638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:10.115291 24638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:10.124577 24646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:10.124632 24645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:10.124585 24656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:10.124815 24638 server_base.cc:1034] running on GCE node
I20260809 09:28:10.125118 24638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:10.125654 24638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:10.126812 24638 hybrid_clock.cc:648] HybridClock initialized: now 1786267690126785 us; error 37 us; skew 500 ppm
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267690, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:10.129408 24638 init.cc:377] Logged in from keytab as oryx/127.9.102.2@KRBTEST.COM (short username oryx)
I20260809 09:28:10.130143 24638 webserver.cc:460] Webserver started at http://127.9.102.2:41673/ using document root <none> and password file <none>
I20260809 09:28:10.130574 24638 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:10.130637 24638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:10.133877 24638 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:28:10.135620 24666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:10.136834 24638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:28:10.136955 24638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "bbb489f8a90c41b5a1cc9fb81fdbab6a"
format_stamp: "Formatted at 2026-08-09 09:28:09 on dist-test-slave-7vjv"
server_key: "37bf36648a1f299f24bd4ebff51c8b81"
server_key_iv: "2ba0c7137797e8e04adaacb3cb9f0011"
server_key_version: "encryptionkey@0"
I20260809 09:28:10.137404 24638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:10.154872 24638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:10.157653 24638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:10.157830 24638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:10.158321 24638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:10.159049 24638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:10.159092 24638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:10.159155 24638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:10.159185 24638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:10.167676 24638 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:37967
I20260809 09:28:10.167698 24843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:37967 every 8 connection(s)
I20260809 09:28:10.168555 24638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:10.177441 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 24638
I20260809 09:28:10.177841 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:38719
--local_ip_for_outbound_sockets=127.9.102.3
--tserver_master_addrs=127.9.102.62:33163
--webserver_port=46705
--webserver_interface=127.9.102.3
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.102.3
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:10.177861 24635 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:10.170197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:33729 (local address 127.9.102.62:33163)
0809 09:28:10.170328 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:10.170333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:10.171209 (+ 876us) server_negotiation.cc:407] Connection header received
0809 09:28:10.171941 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:10.171944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:10.171991 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:10.172058 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:10.173119 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.173569 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.174491 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.174678 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.174749 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.177753 (+ 3004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:33729: BlockingRecv error: recv got EOF from 127.9.102.2:33729 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 09:28:10.178345 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:10.170477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:33729)
0809 09:28:10.171062 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:10.171096 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:10.171780 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:10.172258 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:10.172264 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:10.172618 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:10.172998 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.173009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.173675 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.173677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.174392 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.174400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.174797 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.174800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.174911 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.175361 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:10.175375 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:10.177636 (+ 2261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":104,"threads_started":1}
W20260809 09:28:10.178716 24845 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:33163 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.102.62:33163: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:10.184053 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:10.179091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:41253)
0809 09:28:10.179296 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:10.179307 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:10.179377 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:10.179712 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:10.179716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:10.179857 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:10.179935 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.179940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.180926 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.180928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.181412 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.181418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.181991 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.181994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.182053 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.182452 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:10.182461 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:10.183971 (+ 1510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148}
W20260809 09:28:10.184290 24635 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:10.179181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:41253 (local address 127.9.102.62:33163)
0809 09:28:10.179279 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:10.179282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:10.179345 (+ 63us) server_negotiation.cc:407] Connection header received
0809 09:28:10.179479 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:10.179481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:10.179541 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:10.179657 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:10.180058 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.180809 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.181515 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.181881 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.181993 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.184166 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:41253: BlockingRecv error: recv got EOF from 127.9.102.2:41253 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:10.189199 24849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:10.184540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:35311)
0809 09:28:10.184616 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:10.184629 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:10.184697 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:10.184946 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:10.184948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:10.185074 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:10.185129 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.185133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.186140 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.186142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.186579 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.186585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.187144 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.187146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.187191 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.187575 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:10.187585 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:10.189131 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260809 09:28:10.189419 24635 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:10.184630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:35311 (local address 127.9.102.62:33163)
0809 09:28:10.184740 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:10.184746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:10.184761 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:28:10.184797 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:10.184801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:10.184841 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:10.184927 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:10.185241 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.186035 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.186698 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.187040 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.187148 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.189292 (+ 2144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:35311: BlockingRecv error: recv got EOF from 127.9.102.2:35311 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 09:28:10.189496 24845 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 09:28:10.280284 24850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:10.280591 24850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:10.280632 24850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:10.284170 24850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:10.284210 24850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:10.284256 24850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:28:10.288519 24850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/krb5kdc/oryx_127.9.102.3.keytab
--principal=oryx/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:38719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=46705
--tserver_master_addrs=127.9.102.62:33163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:10.289316 24850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:10.290159 24850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:10.298337 24857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:10.298337 24856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:10.298550 24850 server_base.cc:1034] running on GCE node
W20260809 09:28:10.298341 24860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:10.298976 24850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:10.299520 24850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:10.300681 24850 hybrid_clock.cc:648] HybridClock initialized: now 1786267690300648 us; error 44 us; skew 500 ppm
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267690, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:10.303176 24850 init.cc:377] Logged in from keytab as oryx/127.9.102.3@KRBTEST.COM (short username oryx)
I20260809 09:28:10.303911 24850 webserver.cc:460] Webserver started at http://127.9.102.3:46705/ using document root <none> and password file <none>
I20260809 09:28:10.304457 24850 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:10.304536 24850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:10.307854 24850 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:28:10.309762 24873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:10.310850 24850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 09:28:10.310968 24850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "743c2e89af924a28a4b2c3668f304b3e"
format_stamp: "Formatted at 2026-08-09 09:28:09 on dist-test-slave-7vjv"
server_key: "35958bdb62fdf1e0a98869864e10606e"
server_key_iv: "c1e42f7340edda769b6e1d0dfdad71ff"
server_key_version: "encryptionkey@0"
I20260809 09:28:10.311434 24850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:10.330168 24850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:10.332877 24850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:10.333060 24850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:10.333607 24850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:10.334362 24850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:10.334406 24850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:10.334476 24850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:10.334512 24850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:10.345602 24850 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:38719
I20260809 09:28:10.346745 24850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:28:10.347608 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 24850
I20260809 09:28:10.350289 25072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:38719 every 8 connection(s)
I20260809 09:28:10.358685 24635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:10.348708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48530 (local address 127.9.102.62:33163)
0809 09:28:10.348847 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:10.348853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:10.349175 (+ 322us) server_negotiation.cc:407] Connection header received
0809 09:28:10.349330 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:10.349333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:10.349383 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:10.349472 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:10.350475 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.351133 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.352901 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.353252 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.353339 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.354822 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:10.354828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:10.354850 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:10.354874 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:10.356545 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:10.357011 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:10.357016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:10.357022 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:10.357156 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:10.357437 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:10.357440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:10.357442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:10.357830 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:10.358022 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:10.358289 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 09:28:10.358501 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53,"spinlock_wait_cycles":1920}
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
W20260809 09:28:10.361328 25077 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:10.350916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:34001 (local address 127.9.102.62:33163)
0809 09:28:10.351107 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:10.351111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:10.351894 (+ 783us) server_negotiation.cc:407] Connection header received
0809 09:28:10.353028 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:10.353031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:10.353075 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:10.353178 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:10.355013 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.355747 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.357000 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.357321 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.357419 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.361203 (+ 3784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:34001: BlockingRecv error: recv got EOF from 127.9.102.3:34001 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":58,"threads_started":1}
I20260809 09:28:10.361641 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:10.351026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:34001)
0809 09:28:10.351729 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:10.351764 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:10.352741 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:10.353839 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:10.353849 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:10.354286 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:10.354882 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.354897 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.355852 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.355854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.356887 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.356897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.357464 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.357468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.357603 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.358160 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:10.358179 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:10.360799 (+ 2620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":116,"threads_started":1}
W20260809 09:28:10.362143 25073 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:33163 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.102.62:33163: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:10.366751 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:10.362498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:49559)
0809 09:28:10.362607 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:10.362619 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:10.362685 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:10.362852 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:10.362854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:10.362967 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:10.363042 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.363046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.363686 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.363688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.364264 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.364273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.364648 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.364651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.364730 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.365192 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:10.365202 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:10.366678 (+ 1476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 09:28:10.366920 25077 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:10.362580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:49559 (local address 127.9.102.62:33163)
0809 09:28:10.362659 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:10.362663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:10.362676 (+ 13us) server_negotiation.cc:407] Connection header received
0809 09:28:10.362738 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:10.362740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:10.362767 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:10.362824 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:10.363167 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.363584 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.364372 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.364559 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.364630 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.366816 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:49559: BlockingRecv error: recv got EOF from 127.9.102.3:49559 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:10 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:10.371274 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:10.367198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:41455)
0809 09:28:10.367322 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:10.367336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:10.367418 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:10.367665 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:10.367667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:10.367780 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:10.367839 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.367843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.368537 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.368539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.369004 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:10.369011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.369367 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.369369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:10.369416 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.369752 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:10.369761 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:10.371201 (+ 1440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 09:28:10.371423 25077 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:10.367308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:41455 (local address 127.9.102.62:33163)
0809 09:28:10.367492 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:10.367494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:10.367505 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:28:10.367538 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:10.367540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:10.367570 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:10.367633 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:10.367932 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.368431 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.369105 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:10.369279 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:10.369345 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:10.371294 (+ 1949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:41455: BlockingRecv error: recv got EOF from 127.9.102.3:41455 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20260809 09:28:10.371447 25073 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 09:28:11 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:11 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:11.020828 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:11.015826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:56135)
0809 09:28:11.016070 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:11.016083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:11.016203 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:11.016420 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:11.016422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:11.016582 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:11.016671 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:11.016676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:11.017295 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:11.017297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:11.017793 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:11.017800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:11.018180 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:11.018183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:11.018236 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:11.018813 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:11.018823 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:11.020697 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
W20260809 09:28:11.020989 25081 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:11.015908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:56135 (local address 127.9.102.62:33163)
0809 09:28:11.016182 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:11.016185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:11.016197 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:28:11.016273 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:11.016275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:11.016311 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:11.016381 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:11.016763 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:11.017211 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:11.017885 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:11.018070 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:11.018156 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:11.020846 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:56135: BlockingRecv error: recv got EOF from 127.9.102.1:56135 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":105,"threads_started":1}
Aug 09 09:28:11 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:11 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:11.200659 25082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:11.190190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:48055)
0809 09:28:11.190486 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:11.190501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:11.190608 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:11.190877 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:11.190880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:11.191102 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:11.191208 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:11.191214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:11.192026 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:11.192030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:11.193019 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:11.193031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:11.193653 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:11.193660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:11.193765 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:11.198337 (+ 4572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:11.198348 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:11.200459 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":94,"threads_started":1}
W20260809 09:28:11.200809 25081 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:11.190332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:48055 (local address 127.9.102.62:33163)
0809 09:28:11.190487 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:11.190493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:11.190565 (+ 72us) server_negotiation.cc:407] Connection header received
0809 09:28:11.190712 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:11.190716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:11.190760 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:11.190856 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:11.191336 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:11.191906 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:11.193154 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:11.193498 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:11.193610 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:11.200647 (+ 7037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:48055: BlockingRecv error: recv got EOF from 127.9.102.2:48055 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 09:28:11 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:11 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:11.378103 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:11.372017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:57237)
0809 09:28:11.372255 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:11.372268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:11.372366 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:11.372647 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:11.372649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:11.372877 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:11.372992 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:11.372998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:11.373982 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:11.373985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:11.374558 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:11.374568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:11.375291 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:11.375295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:11.375372 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:11.376040 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:11.376053 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:11.377906 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
W20260809 09:28:11.378136 25081 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:11.372060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:57237 (local address 127.9.102.62:33163)
0809 09:28:11.372225 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:11.372231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:11.372321 (+ 90us) server_negotiation.cc:407] Connection header received
0809 09:28:11.372447 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:11.372449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:11.372531 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:11.372614 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:11.373086 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:11.373883 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:11.374725 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:11.375182 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:11.375278 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:11.378038 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:57237: BlockingRecv error: recv got EOF from 127.9.102.3:57237 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 09 09:28:12 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:12 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:12.026858 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:12.021591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:54437)
0809 09:28:12.021868 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:12.021881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:12.021968 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:12.022227 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:12.022229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:12.022395 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:12.022486 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:12.022491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:12.023183 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:12.023185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:12.023684 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:12.023691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:12.024050 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:12.024052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:12.024175 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:12.024759 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:12.024772 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:12.026715 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":93,"threads_started":1}
W20260809 09:28:12.027011 25087 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:12.021778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:54437 (local address 127.9.102.62:33163)
0809 09:28:12.022016 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:12.022020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:12.022038 (+ 18us) server_negotiation.cc:407] Connection header received
0809 09:28:12.022070 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:12.022072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:12.022109 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:12.022180 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:12.022599 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:12.023079 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:12.023779 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:12.023961 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:12.024026 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:12.026854 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:54437: BlockingRecv error: recv got EOF from 127.9.102.1:54437 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
Aug 09 09:28:12 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:12 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:12.206400 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:12.201323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:54811)
0809 09:28:12.201587 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:12.201602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:12.201696 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:12.201933 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:12.201935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:12.202130 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:12.202214 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:12.202219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:12.202906 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:12.202910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:12.203425 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:12.203432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:12.203876 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:12.203879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:12.203963 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:12.204520 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:12.204530 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:12.206226 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":105,"threads_started":1}
W20260809 09:28:12.206590 25087 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:12.201520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:54811 (local address 127.9.102.62:33163)
0809 09:28:12.201651 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:12.201655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:12.201667 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:28:12.201771 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:12.201774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:12.201824 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:12.201910 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:12.202340 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:12.202796 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:12.203548 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:12.203756 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:12.203863 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:12.206362 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:54811: BlockingRecv error: recv got EOF from 127.9.102.2:54811 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 09 09:28:12 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:12 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:12.384404 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:12.378723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:37649)
0809 09:28:12.379019 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:12.379045 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:12.379173 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:12.379445 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:12.379449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:12.379678 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:12.379773 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:12.379778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:12.380612 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:12.380616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:12.381175 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:12.381183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:12.381656 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:12.381660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:12.381782 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:12.382334 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:12.382347 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:12.384231 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":100,"threads_started":1}
W20260809 09:28:12.384464 25087 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:12.378878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:37649 (local address 127.9.102.62:33163)
0809 09:28:12.379019 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:12.379024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:12.379117 (+ 93us) server_negotiation.cc:407] Connection header received
0809 09:28:12.379269 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:12.379273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:12.379328 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:12.379418 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:12.379894 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:12.380488 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:12.381302 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:12.381530 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:12.381643 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:12.384357 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:37649: BlockingRecv error: recv got EOF from 127.9.102.3:37649 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 09 09:28:13 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:13 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:13.035226 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:13.027683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:35693)
0809 09:28:13.028002 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:13.028014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:13.028115 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:13.028327 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:13.028329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:13.028489 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:13.028593 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:13.028598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:13.029239 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:13.029241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:13.029717 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:13.029724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:13.030097 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:13.030099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:13.030152 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:13.033370 (+ 3218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:13.033381 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:13.035081 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":115,"threads_started":1}
W20260809 09:28:13.035400 25091 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:13.027793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:35693 (local address 127.9.102.62:33163)
0809 09:28:13.028056 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:13.028059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:13.028071 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:28:13.028188 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:13.028190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:13.028227 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:13.028287 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:13.028700 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:13.029153 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:13.029813 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:13.030005 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:13.030071 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:13.035228 (+ 5157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:35693: BlockingRecv error: recv got EOF from 127.9.102.1:35693 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
Aug 09 09:28:13 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:13 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:13.212525 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:13.207076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:38277)
0809 09:28:13.207383 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:13.207395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:13.207485 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:13.207712 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:13.207714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:13.207925 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:13.208063 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:13.208071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:13.209131 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:13.209133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:13.209691 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:13.209699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:13.210101 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:13.210103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:13.210164 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:13.210618 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:13.210629 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:13.212417 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
W20260809 09:28:13.212666 25091 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:13.207178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:38277 (local address 127.9.102.62:33163)
0809 09:28:13.207318 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:13.207321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:13.207456 (+ 135us) server_negotiation.cc:407] Connection header received
0809 09:28:13.207564 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:13.207566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:13.207608 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:13.207677 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:13.208220 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:13.209012 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:13.209787 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:13.210000 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:13.210082 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:13.212546 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:38277: BlockingRecv error: recv got EOF from 127.9.102.2:38277 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 09:28:13 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:13 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:13.390266 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:13.385150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:55883)
0809 09:28:13.385373 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:13.385384 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:13.385477 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:13.385665 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:13.385667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:13.385852 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:13.385943 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:13.385948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:13.386618 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:13.386621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:13.387154 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:13.387162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:13.387549 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:13.387552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:13.387611 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:13.388167 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:13.388182 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:13.390136 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
W20260809 09:28:13.390390 25091 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:13.385229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:55883 (local address 127.9.102.62:33163)
0809 09:28:13.385386 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:13.385390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:13.385448 (+ 58us) server_negotiation.cc:407] Connection header received
0809 09:28:13.385548 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:13.385550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:13.385592 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:13.385653 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:13.386041 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:13.386530 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:13.387250 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:13.387453 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:13.387524 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:13.390271 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:55883: BlockingRecv error: recv got EOF from 127.9.102.3:55883 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 09 09:28:14 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:14 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:14.041018 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:14.036039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:51865)
0809 09:28:14.036351 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:14.036364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:14.036445 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:14.036706 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:14.036708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:14.036874 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:14.036965 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:14.036969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:14.037608 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:14.037610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:14.038077 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:14.038084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:14.038449 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:14.038451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:14.038506 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:14.038960 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:14.038970 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:14.040865 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":113,"threads_started":1}
W20260809 09:28:14.041095 25097 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:14.036134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:51865 (local address 127.9.102.62:33163)
0809 09:28:14.036388 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:14.036392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:14.036406 (+ 14us) server_negotiation.cc:407] Connection header received
0809 09:28:14.036545 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:14.036547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:14.036588 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:14.036669 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:14.037066 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:14.037525 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:14.038169 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:14.038360 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:14.038427 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:14.040979 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:51865: BlockingRecv error: recv got EOF from 127.9.102.1:51865 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":63,"threads_started":1}
Aug 09 09:28:14 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:14 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:14.218633 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:14.213379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:39381)
0809 09:28:14.213665 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:14.213684 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:14.213771 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:14.213974 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:14.213976 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:14.214144 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:14.214235 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:14.214240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:14.214950 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:14.214952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:14.215426 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:14.215433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:14.215888 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:14.215890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:14.215950 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:14.216609 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:14.216621 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:14.218525 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":138,"threads_started":1}
W20260809 09:28:14.218792 25097 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:14.213462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:39381 (local address 127.9.102.62:33163)
0809 09:28:14.213629 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:14.213633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:14.213730 (+ 97us) server_negotiation.cc:407] Connection header received
0809 09:28:14.213843 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:14.213845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:14.213882 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:14.213937 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:14.214339 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:14.214862 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:14.215521 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:14.215794 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:14.215868 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:14.218647 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:39381: BlockingRecv error: recv got EOF from 127.9.102.2:39381 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 09 09:28:14 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:14 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:14.396613 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:14.391294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:33073)
0809 09:28:14.391610 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:14.391621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:14.391744 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:14.392001 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:14.392004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:14.392211 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:14.392300 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:14.392305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:14.392992 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:14.392996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:14.393505 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:14.393512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:14.394004 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:14.394008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:14.394089 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:14.394567 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:14.394578 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:14.396387 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
W20260809 09:28:14.396623 25097 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:14.391400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:33073 (local address 127.9.102.62:33163)
0809 09:28:14.391557 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:14.391561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:14.391701 (+ 140us) server_negotiation.cc:407] Connection header received
0809 09:28:14.391843 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:14.391846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:14.391893 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:14.391972 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:14.392418 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:14.392872 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:14.393654 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:14.393868 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:14.394000 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:14.396492 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:33073: BlockingRecv error: recv got EOF from 127.9.102.3:33073 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 09:28:15 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:15 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:15.047071 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:15.041739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:46883)
0809 09:28:15.042009 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:15.042023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:15.042103 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:15.042336 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:15.042338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:15.042537 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:15.042643 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:15.042648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:15.043296 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:15.043298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:15.043819 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:15.043827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:15.044222 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:15.044225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:15.044280 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:15.044800 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:15.044810 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:15.046915 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
W20260809 09:28:15.047147 25102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:15.041816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:46883 (local address 127.9.102.62:33163)
0809 09:28:15.042049 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:15.042053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:15.042079 (+ 26us) server_negotiation.cc:407] Connection header received
0809 09:28:15.042190 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:15.042192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:15.042231 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:15.042298 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:15.042737 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:15.043217 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:15.043921 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:15.044134 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:15.044203 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:15.047026 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:46883: BlockingRecv error: recv got EOF from 127.9.102.1:46883 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":64,"threads_started":1}
Aug 09 09:28:15 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:15 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:15.224591 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:15.219441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:35747)
0809 09:28:15.219746 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:15.219768 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:15.219868 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:15.220162 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:15.220167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:15.220348 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:15.220443 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:15.220449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:15.221167 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:15.221170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:15.221698 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:15.221705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:15.222068 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:15.222070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:15.222124 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:15.222632 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:15.222643 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:15.224468 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
W20260809 09:28:15.224740 25102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:15.219608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:35747 (local address 127.9.102.62:33163)
0809 09:28:15.219807 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:15.219810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:15.219827 (+ 17us) server_negotiation.cc:407] Connection header received
0809 09:28:15.219954 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:15.219957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:15.219998 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:15.220076 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:15.220550 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:15.221074 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:15.221791 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:15.221977 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:15.222050 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:15.224619 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:35747: BlockingRecv error: recv got EOF from 127.9.102.2:35747 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 09 09:28:15 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:15 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:15.403152 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:15.397636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:36189)
0809 09:28:15.397910 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:15.397923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:15.398013 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:15.398255 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:15.398257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:15.398442 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:15.398561 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:15.398566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:15.399343 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:15.399348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:15.400256 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:15.400272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:15.400713 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:15.400716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:15.400814 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:15.401270 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:15.401281 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:15.403029 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
W20260809 09:28:15.403291 25102 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:15.397730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:36189 (local address 127.9.102.62:33163)
0809 09:28:15.397862 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:15.397865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:15.397992 (+ 127us) server_negotiation.cc:407] Connection header received
0809 09:28:15.398098 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:15.398100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:15.398142 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:15.398210 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:15.398657 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:15.399236 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:15.400369 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:15.400590 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:15.400705 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:15.403160 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:36189: BlockingRecv error: recv got EOF from 127.9.102.3:36189 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 09:28:16 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:16 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:16.053179 25109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:16.047772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:60395)
0809 09:28:16.048049 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:16.048062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:16.048173 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:16.048402 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:16.048404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:16.048586 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:16.048699 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:16.048709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:16.049764 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:16.049768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:16.050308 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:16.050317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:16.050726 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:16.050728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:16.050785 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:16.051224 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:16.051234 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:16.053018 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":108,"threads_started":1}
W20260809 09:28:16.053256 25110 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:16.047830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:60395 (local address 127.9.102.62:33163)
0809 09:28:16.048068 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:16.048071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:16.048148 (+ 77us) server_negotiation.cc:407] Connection header received
0809 09:28:16.048251 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:16.048252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:16.048294 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:16.048365 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:16.048838 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:16.049616 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:16.050412 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:16.050628 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:16.050703 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:16.053136 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:60395: BlockingRecv error: recv got EOF from 127.9.102.1:60395 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":60,"threads_started":1}
Aug 09 09:28:16 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:16 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:16.230113 25112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:16.225266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:56469)
0809 09:28:16.225513 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:16.225532 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:16.225616 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:16.225811 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:16.225813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:16.225975 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:16.226061 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:16.226065 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:16.226669 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:16.226671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:16.227134 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:16.227140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:16.227510 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:16.227512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:16.227591 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:16.228046 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:16.228056 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:16.229963 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":86,"threads_started":1}
W20260809 09:28:16.230211 25110 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:16.225345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:56469 (local address 127.9.102.62:33163)
0809 09:28:16.225481 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:16.225485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:16.225592 (+ 107us) server_negotiation.cc:407] Connection header received
0809 09:28:16.225687 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:16.225689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:16.225726 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:16.225779 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:16.226147 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:16.226588 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:16.227220 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:16.227406 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:16.227490 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:16.230087 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:56469: BlockingRecv error: recv got EOF from 127.9.102.2:56469 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 09:28:16 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:16 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:16.409569 25113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:16.404165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:41865)
0809 09:28:16.404457 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:16.404469 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:16.404617 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:16.404890 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:16.404893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:16.405159 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:16.405247 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:16.405252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:16.406007 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:16.406012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:16.406539 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:16.406547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:16.407003 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:16.407006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:16.407084 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:16.407600 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:16.407611 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:16.409452 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
W20260809 09:28:16.409690 25110 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:16.404221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:41865 (local address 127.9.102.62:33163)
0809 09:28:16.404377 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:16.404382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:16.404556 (+ 174us) server_negotiation.cc:407] Connection header received
0809 09:28:16.404729 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:16.404732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:16.404781 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:16.404866 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:16.405358 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:16.405880 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:16.406670 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:16.406886 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:16.406989 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:16.409579 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:41865: BlockingRecv error: recv got EOF from 127.9.102.3:41865 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 09 09:28:17 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:17 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:17.059852 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:17.054108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:35077)
0809 09:28:17.054752 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:17.054768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:17.054880 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:17.055142 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:17.055145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:17.055353 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:17.055453 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:17.055460 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:17.056197 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:17.056199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:17.056731 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:17.056738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:17.057202 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:17.057205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:17.057268 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:17.057873 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:17.057885 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:17.059737 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":88,"threads_started":1}
W20260809 09:28:17.060007 25114 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:17.054162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:35077 (local address 127.9.102.62:33163)
0809 09:28:17.054438 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:17.054442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:17.054850 (+ 408us) server_negotiation.cc:407] Connection header received
0809 09:28:17.054963 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:17.054965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:17.055011 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:17.055101 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:17.055564 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:17.056100 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:17.056862 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:17.057068 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:17.057140 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:17.059871 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:35077: BlockingRecv error: recv got EOF from 127.9.102.1:35077 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
Aug 09 09:28:17 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:17 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:17.236433 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:17.230877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:36111)
0809 09:28:17.231242 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:17.231258 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:17.231395 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:17.231668 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:17.231670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:17.231866 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:17.231951 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:17.231957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:17.232680 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:17.232683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:17.233171 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:17.233178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:17.233619 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:17.233623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:17.233701 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:17.234158 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:17.234169 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:17.236283 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
W20260809 09:28:17.236478 25114 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:17.231096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:36111 (local address 127.9.102.62:33163)
0809 09:28:17.231243 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:17.231248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:17.231337 (+ 89us) server_negotiation.cc:407] Connection header received
0809 09:28:17.231487 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:17.231490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:17.231542 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:17.231640 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:17.232071 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:17.232570 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:17.233286 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:17.233507 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:17.233616 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:17.236387 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:36111: BlockingRecv error: recv got EOF from 127.9.102.2:36111 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 09 09:28:17 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:17 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:17.415393 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:17.410324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:43913)
0809 09:28:17.410631 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:17.410642 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:17.410731 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:17.410946 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:17.410948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:17.411116 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:17.411204 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:17.411209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:17.411871 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:17.411873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:17.412471 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:17.412491 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:17.412872 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:17.412875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:17.412926 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:17.413363 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:17.413374 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:17.415236 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":111,"threads_started":1}
W20260809 09:28:17.415439 25114 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:17.410394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:43913 (local address 127.9.102.62:33163)
0809 09:28:17.410517 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:17.410521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:17.410705 (+ 184us) server_negotiation.cc:407] Connection header received
0809 09:28:17.410815 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:17.410817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:17.410858 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:17.410909 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:17.411317 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:17.411783 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:17.412594 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:17.412785 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:17.412854 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:17.415344 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:43913: BlockingRecv error: recv got EOF from 127.9.102.3:43913 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 09:28:18 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:18 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:18.068879 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:18.060695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:46967)
0809 09:28:18.060966 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:18.060986 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:18.061075 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:18.061272 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:18.061274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:18.061484 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:18.061595 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:18.061600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:18.062329 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:18.062334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:18.062837 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:18.062846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:18.063295 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:18.063299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:18.063383 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:18.066888 (+ 3505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:18.066899 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:18.068771 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":106,"threads_started":1}
W20260809 09:28:18.069141 25122 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:18.060876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:46967 (local address 127.9.102.62:33163)
0809 09:28:18.061081 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:18.061084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:18.061095 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:28:18.061128 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:18.061130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:18.061166 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:18.061236 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:18.061707 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:18.062207 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:18.062959 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:18.063177 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:18.063280 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:18.069008 (+ 5728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:46967: BlockingRecv error: recv got EOF from 127.9.102.1:46967 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
Aug 09 09:28:18 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:18 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:18.242915 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:18.237309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:38439)
0809 09:28:18.237629 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:18.237644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:18.237758 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:18.238073 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:18.238076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:18.238326 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:18.238435 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:18.238441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:18.239244 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:18.239247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:18.239793 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:18.239800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:18.240210 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:18.240213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:18.240273 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:18.240886 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:18.240896 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:18.242813 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
W20260809 09:28:18.243086 25122 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:18.237418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:38439 (local address 127.9.102.62:33163)
0809 09:28:18.237571 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:18.237575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:18.237731 (+ 156us) server_negotiation.cc:407] Connection header received
0809 09:28:18.237870 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:18.237876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:18.237935 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:18.238011 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:18.238541 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:18.239155 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:18.239893 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:18.240119 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:18.240194 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:18.242939 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:38439: BlockingRecv error: recv got EOF from 127.9.102.2:38439 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 09 09:28:18 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:18 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:18.421710 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:18.416170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:51695)
0809 09:28:18.416411 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:18.416426 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:18.416539 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:18.416760 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:18.416763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:18.416938 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:18.417034 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:18.417040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:18.417879 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:18.417883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:18.418681 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:18.418692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:18.419107 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:18.419110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:18.419168 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:18.419648 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:18.419659 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:18.421599 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":85,"threads_started":1}
W20260809 09:28:18.421852 25122 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:18.416189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:51695 (local address 127.9.102.62:33163)
0809 09:28:18.416325 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:18.416329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:18.416503 (+ 174us) server_negotiation.cc:407] Connection header received
0809 09:28:18.416613 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:18.416615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:18.416659 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:18.416727 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:18.417132 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:18.417757 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:18.418784 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:18.419001 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:18.419089 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:18.421729 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:51695: BlockingRecv error: recv got EOF from 127.9.102.3:51695 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 09:28:19 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:19 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:19.074877 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:19.069528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:38975)
0809 09:28:19.069787 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:19.069799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:19.069893 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:19.070133 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:19.070135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:19.070311 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:19.070405 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:19.070410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:19.071065 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:19.071067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:19.071599 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:19.071606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:19.071996 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:19.071999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:19.072063 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:19.072662 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:19.072672 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:19.074742 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
W20260809 09:28:19.074977 25129 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:19.069590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:38975 (local address 127.9.102.62:33163)
0809 09:28:19.069811 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:19.069814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:19.069861 (+ 47us) server_negotiation.cc:407] Connection header received
0809 09:28:19.069961 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:19.069963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:19.070021 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:19.070097 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:19.070495 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:19.070977 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:19.071691 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:19.071906 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:19.071976 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:19.074866 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:38975: BlockingRecv error: recv got EOF from 127.9.102.1:38975 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":58,"threads_started":1}
Aug 09 09:28:19 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:19 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:19.249135 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:19.243620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:50215)
0809 09:28:19.243888 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:19.243899 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:19.244051 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:19.244345 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:19.244348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:19.244560 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:19.244651 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:19.244656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:19.245457 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:19.245464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:19.246105 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:19.246114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:19.246502 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:19.246505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:19.246613 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:19.247090 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:19.247101 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:19.249023 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":108,"threads_started":1}
W20260809 09:28:19.249305 25129 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:19.243677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:50215 (local address 127.9.102.62:33163)
0809 09:28:19.243815 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:19.243819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:19.243971 (+ 152us) server_negotiation.cc:407] Connection header received
0809 09:28:19.244186 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:19.244189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:19.244238 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:19.244326 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:19.244814 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:19.245347 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:19.246218 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:19.246415 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:19.246487 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:19.249149 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:50215: BlockingRecv error: recv got EOF from 127.9.102.2:50215 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 09:28:19 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:19 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:19.427742 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:19.422601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:59845)
0809 09:28:19.422850 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:19.422861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:19.422947 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:19.423169 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:19.423171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:19.423330 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:19.423419 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:19.423423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:19.424081 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:19.424100 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:19.424731 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:19.424761 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:19.425285 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:19.425289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:19.425367 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:19.425806 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:19.425816 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:19.427598 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":98,"threads_started":1}
W20260809 09:28:19.428000 25129 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:19.422650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:59845 (local address 127.9.102.62:33163)
0809 09:28:19.422761 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:19.422765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:19.422917 (+ 152us) server_negotiation.cc:407] Connection header received
0809 09:28:19.423014 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:19.423016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:19.423062 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:19.423123 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:19.423503 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:19.423978 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:19.424879 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:19.425165 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:19.425270 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:19.427766 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:59845: BlockingRecv error: recv got EOF from 127.9.102.3:59845 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 09:28:20 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:20 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:20.080699 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:20.075785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:35897)
0809 09:28:20.076056 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:20.076068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:20.076172 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:20.076395 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:20.076397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:20.076573 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:20.076664 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:20.076669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:20.077342 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:20.077345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:20.077912 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:20.077920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:20.078296 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:20.078298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:20.078355 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:20.078815 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:20.078826 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:20.080569 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
W20260809 09:28:20.080919 25135 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:20.075825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:35897 (local address 127.9.102.62:33163)
0809 09:28:20.076035 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:20.076039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:20.076143 (+ 104us) server_negotiation.cc:407] Connection header received
0809 09:28:20.076250 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:20.076252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:20.076290 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:20.076360 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:20.076754 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:20.077255 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:20.078006 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:20.078206 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:20.078276 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:20.080699 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:35897: BlockingRecv error: recv got EOF from 127.9.102.1:35897 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":62,"threads_started":1}
Aug 09 09:28:20 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:20 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:20.254833 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:20.249855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:35909)
0809 09:28:20.250101 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:20.250113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:20.250231 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:20.250482 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:20.250485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:20.250674 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:20.250762 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:20.250767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:20.251431 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:20.251435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:20.251920 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:20.251928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:20.252400 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:20.252403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:20.252477 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:20.252971 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:20.252982 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:20.254732 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
W20260809 09:28:20.254998 25135 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:20.249893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:35909 (local address 127.9.102.62:33163)
0809 09:28:20.250027 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:20.250032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:20.250184 (+ 152us) server_negotiation.cc:407] Connection header received
0809 09:28:20.250329 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:20.250333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:20.250381 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:20.250460 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:20.250869 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:20.251329 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:20.252029 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:20.252296 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:20.252396 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:20.254855 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:35909: BlockingRecv error: recv got EOF from 127.9.102.2:35909 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 09 09:28:20 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:20 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:20.432945 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:20.428429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:49411)
0809 09:28:20.428662 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:20.428674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:20.428753 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:20.428956 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:20.428957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:20.429133 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:20.429218 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:20.429222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:20.429845 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:20.429847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:20.430314 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:20.430320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:20.430673 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:20.430675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:20.430729 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:20.431148 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:20.431159 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:20.432834 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":61,"threads_started":1}
W20260809 09:28:20.433074 25135 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:20.428556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:49411 (local address 127.9.102.62:33163)
0809 09:28:20.428720 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:20.428723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:20.428735 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:28:20.428825 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:20.428827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:20.428867 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:20.428926 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:20.429305 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:20.429753 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:20.430403 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:20.430587 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:20.430652 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:20.432964 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:49411: BlockingRecv error: recv got EOF from 127.9.102.3:49411 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 09 09:28:21 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:21 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:21.086925 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:21.081524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:38181)
0809 09:28:21.081824 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:21.081840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:21.081959 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:21.082239 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:21.082242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:21.082477 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:21.082605 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:21.082612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:21.083377 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:21.083382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:21.083913 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:21.083922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:21.084467 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:21.084470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:21.084565 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:21.085072 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:21.085083 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:21.086816 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
W20260809 09:28:21.087078 25142 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:21.081583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:38181 (local address 127.9.102.62:33163)
0809 09:28:21.081865 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:21.081868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:21.081917 (+ 49us) server_negotiation.cc:407] Connection header received
0809 09:28:21.082069 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:21.082072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:21.082124 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:21.082216 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:21.082736 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:21.083265 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:21.084044 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:21.084332 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:21.084448 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:21.086946 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:38181: BlockingRecv error: recv got EOF from 127.9.102.1:38181 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":62,"threads_started":1}
Aug 09 09:28:21 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:21 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:21.260517 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:21.255479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:39183)
0809 09:28:21.255729 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:21.255741 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:21.255832 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:21.256034 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:21.256036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:21.256259 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:21.256365 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:21.256370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:21.257166 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:21.257169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:21.257717 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:21.257725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:21.258092 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:21.258094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:21.258148 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:21.258656 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:21.258667 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:21.260413 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
W20260809 09:28:21.260682 25142 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:21.255558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:39183 (local address 127.9.102.62:33163)
0809 09:28:21.255699 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:21.255702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:21.255791 (+ 89us) server_negotiation.cc:407] Connection header received
0809 09:28:21.255893 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:21.255895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:21.255932 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:21.255997 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:21.256473 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:21.257070 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:21.257812 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:21.258000 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:21.258073 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:21.260567 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:39183: BlockingRecv error: recv got EOF from 127.9.102.2:39183 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 09:28:21 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:21 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:21.440548 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:21.433791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:59089)
0809 09:28:21.434073 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:21.434093 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:21.434231 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:21.434540 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:21.434544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:21.434790 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:21.434974 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:21.434986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:21.436144 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:21.436149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:21.437043 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:21.437058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:21.437535 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:21.437541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:21.437645 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:21.438244 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:21.438262 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:21.440378 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":67,"threads_started":1}
W20260809 09:28:21.440773 25142 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:21.433856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:59089 (local address 127.9.102.62:33163)
0809 09:28:21.433970 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:21.433975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:21.434167 (+ 192us) server_negotiation.cc:407] Connection header received
0809 09:28:21.434342 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:21.434345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:21.434412 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:21.434481 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:21.435109 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:21.435978 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:21.437172 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:21.437440 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:21.437512 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:21.440614 (+ 3102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:59089: BlockingRecv error: recv got EOF from 127.9.102.3:59089 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 09:28:22 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:22 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:22.092877 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:22.087596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:37923)
0809 09:28:22.087864 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:22.087881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:22.087986 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:22.088273 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:22.088276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:22.088524 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:22.088658 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:22.088666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:22.089644 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:22.089647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:22.090150 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:22.090158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:22.090520 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:22.090523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:22.090579 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:22.091094 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:22.091106 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:22.092773 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
W20260809 09:28:22.093019 25154 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:22.087659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:37923 (local address 127.9.102.62:33163)
0809 09:28:22.087906 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:22.087910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:22.087938 (+ 28us) server_negotiation.cc:407] Connection header received
0809 09:28:22.088067 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:22.088069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:22.088142 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:22.088209 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:22.088780 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:22.089543 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:22.090240 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:22.090435 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:22.090501 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:22.092895 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:37923: BlockingRecv error: recv got EOF from 127.9.102.1:37923 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":99,"threads_started":1}
Aug 09 09:28:22 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:22 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:22.266196 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:22.261287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:37559)
0809 09:28:22.261555 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:22.261567 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:22.261704 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:22.261901 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:22.261903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:22.262076 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:22.262165 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:22.262169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:22.262876 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:22.262879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:22.263452 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:22.263459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:22.263848 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:22.263851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:22.263910 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:22.264383 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:22.264393 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:22.266095 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
W20260809 09:28:22.266355 25154 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:22.261338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:37559 (local address 127.9.102.62:33163)
0809 09:28:22.261486 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:22.261490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:22.261668 (+ 178us) server_negotiation.cc:407] Connection header received
0809 09:28:22.261767 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:22.261769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:22.261809 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:22.261865 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:22.262254 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:22.262758 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:22.263552 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:22.263757 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:22.263829 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:22.266218 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:37559: BlockingRecv error: recv got EOF from 127.9.102.2:37559 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 09:28:22 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:22 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:22.446199 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:22.441257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:52033)
0809 09:28:22.441495 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:22.441507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:22.441601 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:22.441794 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:22.441796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:22.442016 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:22.442120 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:22.442126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:22.442785 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:22.442787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:22.443293 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:22.443300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:22.443722 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:22.443724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:22.443778 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:22.444225 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:22.444236 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:22.446065 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
W20260809 09:28:22.446401 25154 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:22.441331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:52033 (local address 127.9.102.62:33163)
0809 09:28:22.441453 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:22.441457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:22.441558 (+ 101us) server_negotiation.cc:407] Connection header received
0809 09:28:22.441661 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:22.441663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:22.441701 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:22.441761 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:22.442222 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:22.442699 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:22.443395 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:22.443634 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:22.443704 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:22.446244 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:52033: BlockingRecv error: recv got EOF from 127.9.102.3:52033 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 09:28:23 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:23 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:23.102376 25164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:23.093683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:44705)
0809 09:28:23.093952 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:23.093965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:23.094056 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:23.094304 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:23.094307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:23.094491 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:23.094599 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:23.094604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:23.095605 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:23.095608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:23.096202 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:23.096211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:23.096608 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:23.096611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:23.096665 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:23.100480 (+ 3815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:23.100491 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:23.102211 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
W20260809 09:28:23.102447 25166 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:23.093815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:44705 (local address 127.9.102.62:33163)
0809 09:28:23.094081 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:23.094084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:23.094099 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:28:23.094137 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:23.094139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:23.094181 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:23.094270 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:23.094712 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:23.095463 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:23.096315 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:23.096512 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:23.096593 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:23.102327 (+ 5734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:44705: BlockingRecv error: recv got EOF from 127.9.102.1:44705 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":93,"threads_started":1}
Aug 09 09:28:23 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:23 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:23.272742 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:23.266934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:47031)
0809 09:28:23.267249 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:23.267265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:23.267399 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:23.267658 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:23.267661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:23.267863 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:23.267951 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:23.267955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:23.268808 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:23.268814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:23.269349 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:23.269358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:23.269784 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:23.269788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:23.269912 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:23.270459 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:23.270471 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:23.272572 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":122,"threads_started":1}
W20260809 09:28:23.272866 25166 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:23.267090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:47031 (local address 127.9.102.62:33163)
0809 09:28:23.267251 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:23.267257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:23.267341 (+ 84us) server_negotiation.cc:407] Connection header received
0809 09:28:23.267490 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:23.267493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:23.267549 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:23.267633 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:23.268063 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:23.268621 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:23.269470 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:23.269675 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:23.269781 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:23.272717 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:47031: BlockingRecv error: recv got EOF from 127.9.102.2:47031 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 09:28:23 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:23 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:23.452246 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:23.446964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:44227)
0809 09:28:23.447245 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:23.447261 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:23.447366 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:23.447633 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:23.447636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:23.447852 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:23.447952 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:23.447957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:23.448700 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:23.448705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:23.449225 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:23.449233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:23.449668 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:23.449672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:23.449766 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:23.450235 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:23.450246 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:23.452056 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
W20260809 09:28:23.452344 25166 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:23.447011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:44227 (local address 127.9.102.62:33163)
0809 09:28:23.447166 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:23.447170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:23.447315 (+ 145us) server_negotiation.cc:407] Connection header received
0809 09:28:23.447462 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:23.447465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:23.447517 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:23.447606 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:23.448067 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:23.448594 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:23.449341 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:23.449561 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:23.449656 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:23.452225 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:44227: BlockingRecv error: recv got EOF from 127.9.102.3:44227 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 09:28:24 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:24 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:24.109169 25169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:24.103299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:57043)
0809 09:28:24.103640 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:24.103682 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:24.103812 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:24.104094 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:24.104096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:24.104400 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:24.104513 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:24.104520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:24.105255 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:24.105258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:24.105889 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:24.105897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:24.106315 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:24.106318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:24.106385 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:24.106988 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:24.107001 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:24.109040 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":121,"threads_started":1}
W20260809 09:28:24.109349 25170 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:24.103338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:57043 (local address 127.9.102.62:33163)
0809 09:28:24.103714 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:24.103719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:24.103736 (+ 17us) server_negotiation.cc:407] Connection header received
0809 09:28:24.103921 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:24.103923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:24.103967 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:24.104072 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:24.104628 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:24.105157 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:24.105994 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:24.106215 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:24.106291 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:24.109177 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:57043: BlockingRecv error: recv got EOF from 127.9.102.1:57043 (error 108)
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":75,"threads_started":1}
Aug 09 09:28:24 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:24 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:24.278687 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:24.273497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:39467)
0809 09:28:24.273771 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:24.273783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:24.273903 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:24.274161 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:24.274164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:24.274369 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:24.274458 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:24.274463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:24.275169 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:24.275173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:24.275693 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:24.275701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:24.276152 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:24.276155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:24.276239 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:24.276747 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:24.276758 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:24.278522 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
W20260809 09:28:24.278735 25170 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:24.273563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:39467 (local address 127.9.102.62:33163)
0809 09:28:24.273698 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:24.273702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:24.273853 (+ 151us) server_negotiation.cc:407] Connection header received
0809 09:28:24.274002 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:24.274005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:24.274061 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:24.274141 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:24.274575 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:24.275067 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:24.275808 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:24.276018 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:24.276147 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:24.278640 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:39467: BlockingRecv error: recv got EOF from 127.9.102.2:39467 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 09:28:24 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:24 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.3@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:24.458391 25175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:24.452910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.3:35951)
0809 09:28:24.453183 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:24.453195 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:24.453282 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:24.453501 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:24.453503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:24.453680 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:24.453772 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:24.453776 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:24.454423 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:24.454425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:24.455184 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:24.455198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:24.455761 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:24.455764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:24.455850 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:24.456490 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:24.456510 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:24.458285 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":110,"threads_started":1}
W20260809 09:28:24.458506 25170 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:24.453092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:35951 (local address 127.9.102.62:33163)
0809 09:28:24.453212 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:24.453215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:24.453241 (+ 26us) server_negotiation.cc:407] Connection header received
0809 09:28:24.453360 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:24.453362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:24.453402 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:24.453468 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:24.453864 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:24.454337 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:24.455313 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:24.455600 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:24.455744 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:24.458396 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.3:35951: BlockingRecv error: recv got EOF from 127.9.102.3:35951 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 09 09:28:25 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:25 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.1@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:25.115638 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:25.110248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.1:50015)
0809 09:28:25.110727 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:25.110741 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:25.110871 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:25.111190 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:25.111193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:25.111455 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:25.111558 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:25.111565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.112299 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.112301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:25.112833 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:25.112840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.113213 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.113215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:25.113272 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:25.113705 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:25.113715 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:25.115521 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":81,"threads_started":1}
W20260809 09:28:25.115794 25178 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:25.110317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:50015 (local address 127.9.102.62:33163)
0809 09:28:25.110623 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:25.110628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:25.110819 (+ 191us) server_negotiation.cc:407] Connection header received
0809 09:28:25.110972 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:25.110975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:25.111049 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:25.111144 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:25.111674 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.112209 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.112924 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.113124 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.113194 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:25.115641 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.1:50015: BlockingRecv error: recv got EOF from 127.9.102.1:50015 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
Aug 09 09:28:25 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
Aug 09 09:28:25 dist-test-slave-7vjv krb5kdc[23422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.102.2@KRBTEST.COM for oryx/127.9.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260809 09:28:25.284742 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:25.279682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:33163 (local address 127.9.102.2:57361)
0809 09:28:25.279961 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:25.279973 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:25.280101 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:25.280411 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:25.280414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:25.280643 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:25.280749 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:25.280756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.281435 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.281437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:25.281936 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:25.281943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.282307 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.282309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:25.282368 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:25.282809 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:25.282819 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:25.284638 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.102.62:33163: Server oryx/127.9.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":74,"threads_started":1}
W20260809 09:28:25.284858 25178 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:25.279730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:57361 (local address 127.9.102.62:33163)
0809 09:28:25.279851 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:25.279857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:25.280052 (+ 195us) server_negotiation.cc:407] Connection header received
0809 09:28:25.280222 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:25.280226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:25.280280 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:25.280358 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:25.280867 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.281346 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.282032 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.282219 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.282286 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:25.284751 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.102.2:57361: BlockingRecv error: recv got EOF from 127.9.102.2:57361 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 09:28:25.358289 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 24385
I20260809 09:28:25.381256 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 24638
I20260809 09:28:25.390865 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 24850
I20260809 09:28:25.398794 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 24284
2026-08-09T09:28:25Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17968 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T09:28:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:28:25Z Disabled control of system clock
I20260809 09:28:25.434854 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:41229
--webserver_interface=127.9.102.62
--webserver_port=0
--builtin_ntp_servers=127.9.102.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:41229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 09:28:25.538475 25187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:25.538679 25187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:25.538710 25187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:25.542165 25187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:28:25.542210 25187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:25.542224 25187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:25.542238 25187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:28:25.542250 25187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:28:25.546092 25187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:41229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:41229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:25.546813 25187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:25.547842 25187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:25.555974 25200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:25.556017 25198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:25.556150 25203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:25.555997 25187 server_base.cc:1034] running on GCE node
I20260809 09:28:25.556694 25187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:25.557658 25187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:25.558853 25187 hybrid_clock.cc:648] HybridClock initialized: now 1786267705558828 us; error 46 us; skew 500 ppm
I20260809 09:28:25.560660 25187 webserver.cc:460] Webserver started at http://127.9.102.62:39717/ using document root <none> and password file <none>
I20260809 09:28:25.561203 25187 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:25.561295 25187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:25.561509 25187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:25.563270 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "88c71b0e558e40e38363443aa104e3d9"
format_stamp: "Formatted at 2026-08-09 09:28:25 on dist-test-slave-7vjv"
server_key: "db92f6d868feb7f984e578df5c53ca2a"
server_key_iv: "595ce2915cbc30d06eb8d9cb5b5cee83"
server_key_version: "encryptionkey@0"
I20260809 09:28:25.563781 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "88c71b0e558e40e38363443aa104e3d9"
format_stamp: "Formatted at 2026-08-09 09:28:25 on dist-test-slave-7vjv"
server_key: "db92f6d868feb7f984e578df5c53ca2a"
server_key_iv: "595ce2915cbc30d06eb8d9cb5b5cee83"
server_key_version: "encryptionkey@0"
I20260809 09:28:25.567155 25187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 09:28:25.569312 25217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:25.570478 25187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:28:25.570609 25187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "88c71b0e558e40e38363443aa104e3d9"
format_stamp: "Formatted at 2026-08-09 09:28:25 on dist-test-slave-7vjv"
server_key: "db92f6d868feb7f984e578df5c53ca2a"
server_key_iv: "595ce2915cbc30d06eb8d9cb5b5cee83"
server_key_version: "encryptionkey@0"
I20260809 09:28:25.570708 25187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:25.593422 25187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:25.593954 25187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:25.594130 25187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:25.600447 25187 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:41229
I20260809 09:28:25.600463 25301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:41229 every 8 connection(s)
I20260809 09:28:25.601230 25187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:28:25.603653 25302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:25.608436 25302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9: Bootstrap starting.
I20260809 09:28:25.610288 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 25187
I20260809 09:28:25.610416 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:28:25.610613 25302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:25.610706 9624 external_mini_cluster.cc:1428] Setting key f1b8dcf242d49dd3aecf52f57679e000
I20260809 09:28:25.611312 25302 log.cc:824] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9: Log is configured to *not* fsync() on all Append() calls
I20260809 09:28:25.613370 25302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9: No bootstrap required, opened a new log
I20260809 09:28:25.617434 25302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c71b0e558e40e38363443aa104e3d9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 41229 } }
I20260809 09:28:25.617686 25302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:25.617770 25302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c71b0e558e40e38363443aa104e3d9, State: Initialized, Role: FOLLOWER
I20260809 09:28:25.618191 25302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c71b0e558e40e38363443aa104e3d9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 41229 } }
I20260809 09:28:25.618301 25302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:28:25.618373 25302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:28:25.618459 25302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:25.619282 25302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c71b0e558e40e38363443aa104e3d9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 41229 } }
I20260809 09:28:25.619637 25302 leader_election.cc:302] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88c71b0e558e40e38363443aa104e3d9; no voters:
I20260809 09:28:25.619925 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:25.611726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39534 (local address 127.9.102.62:41229)
0809 09:28:25.612368 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:25.612386 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:25.612422 (+ 36us) server_negotiation.cc:407] Connection header received
0809 09:28:25.613125 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:25.613146 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:25.613502 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:25.613817 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:25.614199 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.615612 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.616534 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.617232 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.617369 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:25.618142 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:25.618146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:25.618157 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:28:25.618595 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:25.618653 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:28:25.618672 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:25.618770 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:25.618912 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 09:28:25.619139 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":121,"threads_started":1}
I20260809 09:28:25.619928 25302 leader_election.cc:288] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:28:25.620007 25311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:28:25.620232 25311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [term 1 LEADER]: Becoming Leader. State: Replica: 88c71b0e558e40e38363443aa104e3d9, State: Running, Role: LEADER
I20260809 09:28:25.620571 25311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c71b0e558e40e38363443aa104e3d9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 41229 } }
I20260809 09:28:25.621177 25302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:28:25.622201 25319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88c71b0e558e40e38363443aa104e3d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c71b0e558e40e38363443aa104e3d9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 41229 } } }
I20260809 09:28:25.622339 25319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:25.622665 25327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:28:25.622661 25317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88c71b0e558e40e38363443aa104e3d9. Latest consensus state: current_term: 1 leader_uuid: "88c71b0e558e40e38363443aa104e3d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c71b0e558e40e38363443aa104e3d9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 41229 } } }
I20260809 09:28:25.623610 25317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9 [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:25.625486 25327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:28:25.629936 25327 catalog_manager.cc:1347] Generated new cluster ID: 92de6ece1463490ba7fa68bd33b26e2f
I20260809 09:28:25.629990 25327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:28:25.656181 25327 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:28:25.656893 25327 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:28:25.666260 25327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88c71b0e558e40e38363443aa104e3d9: Generated new TSK 0
I20260809 09:28:25.666859 25327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:28:25.678236 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:41229
--builtin_ntp_servers=127.9.102.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 09:28:25.782788 25344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:25.783005 25344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:25.783061 25344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:25.786696 25344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:25.786738 25344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:25.786783 25344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:28:25.790864 25344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:25.791668 25344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:25.792824 25344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:25.801249 25355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:25.801276 25358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:25.801299 25356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:25.801555 25344 server_base.cc:1034] running on GCE node
I20260809 09:28:25.801944 25344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:25.802464 25344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:25.803668 25344 hybrid_clock.cc:648] HybridClock initialized: now 1786267705803655 us; error 33 us; skew 500 ppm
I20260809 09:28:25.805333 25344 webserver.cc:460] Webserver started at http://127.9.102.1:42689/ using document root <none> and password file <none>
I20260809 09:28:25.805876 25344 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:25.805966 25344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:25.806181 25344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:25.808054 25344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "543bd8f033114a08ae50f1a2e6bd3c98"
format_stamp: "Formatted at 2026-08-09 09:28:25 on dist-test-slave-7vjv"
server_key: "bf7ce141131a026d8357ea86ec66e7b2"
server_key_iv: "19cea5078aaab1b8588e19c45ca1d0b9"
server_key_version: "encryptionkey@0"
I20260809 09:28:25.808640 25344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "543bd8f033114a08ae50f1a2e6bd3c98"
format_stamp: "Formatted at 2026-08-09 09:28:25 on dist-test-slave-7vjv"
server_key: "bf7ce141131a026d8357ea86ec66e7b2"
server_key_iv: "19cea5078aaab1b8588e19c45ca1d0b9"
server_key_version: "encryptionkey@0"
I20260809 09:28:25.811856 25344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 09:28:25.814137 25366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:25.815203 25344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:28:25.815290 25344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "543bd8f033114a08ae50f1a2e6bd3c98"
format_stamp: "Formatted at 2026-08-09 09:28:25 on dist-test-slave-7vjv"
server_key: "bf7ce141131a026d8357ea86ec66e7b2"
server_key_iv: "19cea5078aaab1b8588e19c45ca1d0b9"
server_key_version: "encryptionkey@0"
I20260809 09:28:25.815362 25344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:25.826929 25344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:25.827390 25344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:25.827504 25344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:25.827895 25344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:25.828714 25344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:25.828756 25344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:25.828787 25344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:25.828806 25344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:25.836524 25344 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:36489
I20260809 09:28:25.836580 25544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:36489 every 8 connection(s)
I20260809 09:28:25.837268 25344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:28:25.844712 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:25.838783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:52821 (local address 127.9.102.62:41229)
0809 09:28:25.838917 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:25.838921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:25.839807 (+ 886us) server_negotiation.cc:407] Connection header received
0809 09:28:25.841034 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:25.841037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:25.841079 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:25.841145 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:25.841882 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.842354 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.843286 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.843476 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.843551 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:25.844370 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:25.844372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:25.844374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:28:25.844384 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:25.844400 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:28:25.844406 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:25.844457 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:25.844576 (+ 119us) server_negotiation.cc:299] Negotiation successful
0809 09:28:25.844616 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 09:28:25.845151 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 25344
I20260809 09:28:25.845250 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:28:25.845513 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:25.839127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:41229 (local address 127.9.102.1:52821)
0809 09:28:25.839679 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:25.839711 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:25.840847 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:25.841279 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:25.841284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:25.841352 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:25.841759 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:25.841769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.842449 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.842451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:25.843188 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:25.843196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:25.843588 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:25.843591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:25.843704 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:25.844172 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:28:25.844173 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:25.844203 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:28:25.844251 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:25.844470 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:25.844476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:25.844479 (+ 3us) client_negotiation.cc:769] Sending connection context
0809 09:28:25.844558 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 09:28:25.844724 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
I20260809 09:28:25.845512 9624 external_mini_cluster.cc:1428] Setting key 9556cb6b39302847a97dc0acc64ccd98
I20260809 09:28:25.846513 25545 heartbeater.cc:347] Connected to a master server at 127.9.102.62:41229
I20260809 09:28:25.846728 25545 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:25.847185 25545 heartbeater.cc:511] Master 127.9.102.62:41229 requested a full tablet report, sending...
I20260809 09:28:25.847400 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:41229
--builtin_ntp_servers=127.9.102.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 09:28:25.848472 25240 ts_manager.cc:194] Registered new tserver with Master: 543bd8f033114a08ae50f1a2e6bd3c98 (127.9.102.1:36489)
I20260809 09:28:25.849488 25240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.102.1:52821
W20260809 09:28:25.946278 25548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:25.946470 25548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:25.946516 25548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:25.949978 25548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:25.950014 25548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:25.950059 25548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:28:25.954113 25548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:25.954859 25548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:25.955897 25548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:25.964241 25562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:25.964247 25559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:25.964378 25548 server_base.cc:1034] running on GCE node
W20260809 09:28:25.964536 25556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:25.964927 25548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:25.965485 25548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:25.966646 25548 hybrid_clock.cc:648] HybridClock initialized: now 1786267705966619 us; error 37 us; skew 500 ppm
I20260809 09:28:25.968343 25548 webserver.cc:460] Webserver started at http://127.9.102.2:44965/ using document root <none> and password file <none>
I20260809 09:28:25.968881 25548 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:25.968942 25548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:25.969156 25548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:25.971079 25548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "9689bb9afe3b45bd846f43304b34cf1d"
format_stamp: "Formatted at 2026-08-09 09:28:25 on dist-test-slave-7vjv"
server_key: "5b2a9c81b71259988a910c345b96cd5f"
server_key_iv: "53bf5544d397e5f2aac3fa0b89f3b4d4"
server_key_version: "encryptionkey@0"
I20260809 09:28:25.971828 25548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "9689bb9afe3b45bd846f43304b34cf1d"
format_stamp: "Formatted at 2026-08-09 09:28:25 on dist-test-slave-7vjv"
server_key: "5b2a9c81b71259988a910c345b96cd5f"
server_key_iv: "53bf5544d397e5f2aac3fa0b89f3b4d4"
server_key_version: "encryptionkey@0"
I20260809 09:28:25.976831 25548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260809 09:28:25.980882 25574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:25.982228 25548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:28:25.982323 25548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "9689bb9afe3b45bd846f43304b34cf1d"
format_stamp: "Formatted at 2026-08-09 09:28:25 on dist-test-slave-7vjv"
server_key: "5b2a9c81b71259988a910c345b96cd5f"
server_key_iv: "53bf5544d397e5f2aac3fa0b89f3b4d4"
server_key_version: "encryptionkey@0"
I20260809 09:28:25.982398 25548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:25.997354 25548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:25.997839 25548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:25.998001 25548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:25.998466 25548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:25.999276 25548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:25.999321 25548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:25.999387 25548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:25.999431 25548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.007561 25548 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:33867
I20260809 09:28:26.007592 25768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:33867 every 8 connection(s)
I20260809 09:28:26.008438 25548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:28:26.013819 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 25548
I20260809 09:28:26.013912 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:28:26.014242 9624 external_mini_cluster.cc:1428] Setting key 7100b6ab9d3873b2a0bb261e71bce775
I20260809 09:28:26.016362 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:41229
--builtin_ntp_servers=127.9.102.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 09:28:26.017441 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:26.010113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:52413 (local address 127.9.102.62:41229)
0809 09:28:26.010245 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:26.010248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:26.011058 (+ 810us) server_negotiation.cc:407] Connection header received
0809 09:28:26.012363 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:26.012367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:26.012432 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:26.012519 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:26.013368 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:26.013989 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:26.015453 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:26.015758 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:26.015875 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:26.016906 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:26.016912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:26.016918 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:28:26.016934 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:26.016960 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:28:26.016972 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:26.017060 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:26.017188 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 09:28:26.017252 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 09:28:26.018287 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:26.010348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:41229 (local address 127.9.102.2:52413)
0809 09:28:26.010917 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:26.010951 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:26.012167 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:26.012680 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:26.012686 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:26.012759 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:26.013188 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:26.013218 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:26.014129 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:26.014133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:26.015310 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:26.015320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:26.015918 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:26.015920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:26.016053 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:26.016673 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:28:26.016675 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:26.016708 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:28:26.016763 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:26.017067 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:26.017074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:26.017078 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 09:28:26.017161 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 09:28:26.017344 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":114,"threads_started":1}
I20260809 09:28:26.019387 25770 heartbeater.cc:347] Connected to a master server at 127.9.102.62:41229
I20260809 09:28:26.019647 25770 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:26.020226 25770 heartbeater.cc:511] Master 127.9.102.62:41229 requested a full tablet report, sending...
I20260809 09:28:26.021268 25240 ts_manager.cc:194] Registered new tserver with Master: 9689bb9afe3b45bd846f43304b34cf1d (127.9.102.2:33867)
I20260809 09:28:26.021885 25240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.102.2:52413
W20260809 09:28:26.117089 25774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:26.117280 25774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:26.117309 25774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:26.121206 25774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:26.121248 25774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:26.121315 25774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:28:26.125756 25774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:41229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:26.126574 25774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:26.127645 25774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:26.135757 25785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:26.135792 25782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:26.135766 25783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:26.135967 25774 server_base.cc:1034] running on GCE node
I20260809 09:28:26.136443 25774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:26.136961 25774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:26.138135 25774 hybrid_clock.cc:648] HybridClock initialized: now 1786267706138111 us; error 37 us; skew 500 ppm
I20260809 09:28:26.139760 25774 webserver.cc:460] Webserver started at http://127.9.102.3:44897/ using document root <none> and password file <none>
I20260809 09:28:26.140323 25774 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:26.140424 25774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:26.140642 25774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:26.142385 25774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "98c998af6c4d439a828857b2068c440a"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "078b68b7d1c8181c33a83350d6e69e3e"
server_key_iv: "433efc5e7dd6cfbacd64ab348776b498"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.142884 25774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "98c998af6c4d439a828857b2068c440a"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "078b68b7d1c8181c33a83350d6e69e3e"
server_key_iv: "433efc5e7dd6cfbacd64ab348776b498"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.146018 25774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:28:26.148016 25798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.149058 25774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:28:26.149178 25774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "98c998af6c4d439a828857b2068c440a"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "078b68b7d1c8181c33a83350d6e69e3e"
server_key_iv: "433efc5e7dd6cfbacd64ab348776b498"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.149276 25774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:26.161542 25774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:26.162015 25774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:26.162182 25774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:26.162631 25774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:26.163400 25774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:26.163475 25774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.163532 25774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:26.163561 25774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.171452 25774 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:40207
I20260809 09:28:26.171547 25982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:40207 every 8 connection(s)
I20260809 09:28:26.172495 25774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:28:26.173964 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 25774
I20260809 09:28:26.174065 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:28:26.174360 9624 external_mini_cluster.cc:1428] Setting key 2da1429dfbe232361982197afcccb414
I20260809 09:28:26.180847 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:26.174714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:58913 (local address 127.9.102.62:41229)
0809 09:28:26.174836 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:26.174839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:26.175475 (+ 636us) server_negotiation.cc:407] Connection header received
0809 09:28:26.176858 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:26.176861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:26.176912 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:26.176983 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:26.177708 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:26.178194 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:26.179170 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:26.179370 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:26.179464 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:26.180395 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:26.180398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:26.180401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:28:26.180412 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:26.180431 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:28:26.180445 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:26.180514 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:26.180664 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 09:28:26.180715 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 09:28:26.181500 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:26.174687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:41229 (local address 127.9.102.3:58913)
0809 09:28:26.175357 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:26.175390 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:26.176680 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:26.177128 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:26.177135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:26.177203 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:26.177584 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:26.177595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:26.178312 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:26.178315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:26.179057 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:26.179066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:26.179508 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:26.179511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:26.179697 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:26.180179 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:28:26.180180 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:26.180212 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:28:26.180264 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:26.180528 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:26.180544 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:26.180549 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 09:28:26.180663 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 09:28:26.180913 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":120,"threads_started":1}
I20260809 09:28:26.182379 25983 heartbeater.cc:347] Connected to a master server at 127.9.102.62:41229
I20260809 09:28:26.182611 25983 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:26.183014 25983 heartbeater.cc:511] Master 127.9.102.62:41229 requested a full tablet report, sending...
I20260809 09:28:26.183813 25240 ts_manager.cc:194] Registered new tserver with Master: 98c998af6c4d439a828857b2068c440a (127.9.102.3:40207)
I20260809 09:28:26.184454 25240 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.102.3:58913
I20260809 09:28:26.187717 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 09:28:26.190267 25310 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:26.189464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.9.102.62:41229)
0809 09:28:26.189576 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:26.189581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:26.189760 (+ 179us) server_negotiation.cc:407] Connection header received
0809 09:28:26.189840 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:26.189842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:26.189873 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:26.189929 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:26.190152 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39540: BlockingRecv error: recv got EOF from 127.0.0.1:39540 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 09:28:26.190351 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 25344
I20260809 09:28:26.206563 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 25548
I20260809 09:28:26.222589 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 25774
I20260809 09:28:26.233814 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 25187
2026-08-09T09:28:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (832 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T09:28:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:28:26Z Disabled control of system clock
I20260809 09:28:26.267325 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:35097
--webserver_interface=127.9.102.62
--webserver_port=0
--builtin_ntp_servers=127.9.102.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:35097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 09:28:26.368876 26001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:26.369071 26001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:26.369100 26001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:26.372421 26001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:28:26.372464 26001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:26.372478 26001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:26.372491 26001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:28:26.372536 26001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:28:26.376477 26001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:35097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:35097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:26.377244 26001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:26.378304 26001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:26.385210 26012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:26.385218 26010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:26.385224 26009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:26.385572 26001 server_base.cc:1034] running on GCE node
I20260809 09:28:26.386077 26001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:26.386972 26001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:26.388162 26001 hybrid_clock.cc:648] HybridClock initialized: now 1786267706388140 us; error 28 us; skew 500 ppm
I20260809 09:28:26.389665 26001 webserver.cc:460] Webserver started at http://127.9.102.62:37515/ using document root <none> and password file <none>
I20260809 09:28:26.390195 26001 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:26.390255 26001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:26.390486 26001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:26.392256 26001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "480a8137165f44ffa0e08a623abdc499"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "8011526a2897c1352b0489128f7d9b8c"
server_key_iv: "c5e2f15e7f7a53294ed4fa786b7594d5"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.392791 26001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "480a8137165f44ffa0e08a623abdc499"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "8011526a2897c1352b0489128f7d9b8c"
server_key_iv: "c5e2f15e7f7a53294ed4fa786b7594d5"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.395859 26001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 09:28:26.398018 26021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.398978 26001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 09:28:26.399102 26001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "480a8137165f44ffa0e08a623abdc499"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "8011526a2897c1352b0489128f7d9b8c"
server_key_iv: "c5e2f15e7f7a53294ed4fa786b7594d5"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.399196 26001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:26.403062 26001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:26.403573 26001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:26.403748 26001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:26.410176 26001 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:35097
I20260809 09:28:26.410192 26104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:35097 every 8 connection(s)
I20260809 09:28:26.410930 26001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:28:26.411994 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 26001
I20260809 09:28:26.412127 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:28:26.412374 9624 external_mini_cluster.cc:1428] Setting key aa3b784002bdeb1f012ea338a557b1a6
I20260809 09:28:26.414067 26105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:26.417822 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:26.413624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.9.102.62:35097)
0809 09:28:26.414298 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:26.414317 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:26.414355 (+ 38us) server_negotiation.cc:407] Connection header received
0809 09:28:26.415171 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:26.415192 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:26.415571 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:26.415915 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:26.416558 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:26.416564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:26.416576 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:28:26.416586 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:26.416642 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:28:26.416663 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:26.416798 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:26.416879 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 09:28:26.417091 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":122,"threads_started":1}
I20260809 09:28:26.420177 26105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499: Bootstrap starting.
I20260809 09:28:26.422006 26105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:26.422761 26105 log.cc:824] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499: Log is configured to *not* fsync() on all Append() calls
I20260809 09:28:26.424561 26105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499: No bootstrap required, opened a new log
I20260809 09:28:26.427350 26105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480a8137165f44ffa0e08a623abdc499" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 35097 } }
I20260809 09:28:26.427520 26105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:26.427600 26105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 480a8137165f44ffa0e08a623abdc499, State: Initialized, Role: FOLLOWER
I20260809 09:28:26.427942 26105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480a8137165f44ffa0e08a623abdc499" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 35097 } }
I20260809 09:28:26.428069 26105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:28:26.428144 26105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:28:26.428256 26105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:26.428985 26105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480a8137165f44ffa0e08a623abdc499" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 35097 } }
I20260809 09:28:26.429303 26105 leader_election.cc:302] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 480a8137165f44ffa0e08a623abdc499; no voters:
I20260809 09:28:26.429515 26105 leader_election.cc:288] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:28:26.429591 26115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:28:26.429812 26115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [term 1 LEADER]: Becoming Leader. State: Replica: 480a8137165f44ffa0e08a623abdc499, State: Running, Role: LEADER
I20260809 09:28:26.430161 26115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480a8137165f44ffa0e08a623abdc499" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 35097 } }
I20260809 09:28:26.430608 26105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:28:26.431615 26118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 480a8137165f44ffa0e08a623abdc499. Latest consensus state: current_term: 1 leader_uuid: "480a8137165f44ffa0e08a623abdc499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480a8137165f44ffa0e08a623abdc499" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 35097 } } }
I20260809 09:28:26.431735 26118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:26.432003 26116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "480a8137165f44ffa0e08a623abdc499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "480a8137165f44ffa0e08a623abdc499" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 35097 } } }
I20260809 09:28:26.432086 26116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499 [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:26.432133 26126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:28:26.434705 26126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:28:26.439325 26126 catalog_manager.cc:1347] Generated new cluster ID: f83b68f0c6204b87aec7e5f6a2a7352c
I20260809 09:28:26.439383 26126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:28:26.449833 26126 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:28:26.450088 26126 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:28:26.454504 26126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 480a8137165f44ffa0e08a623abdc499: Generated new TSK 0
I20260809 09:28:26.455025 26126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:28:26.464282 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:35097
--builtin_ntp_servers=127.9.102.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 09:28:26.564769 26140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:26.564980 26140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:26.565047 26140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:26.568811 26140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:26.568853 26140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:26.568931 26140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:28:26.573032 26140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:26.573820 26140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:26.574865 26140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:26.583143 26154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:26.583133 26152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:26.583129 26151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:26.583179 26140 server_base.cc:1034] running on GCE node
I20260809 09:28:26.583738 26140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:26.584280 26140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:26.585427 26140 hybrid_clock.cc:648] HybridClock initialized: now 1786267706585413 us; error 27 us; skew 500 ppm
I20260809 09:28:26.587056 26140 webserver.cc:460] Webserver started at http://127.9.102.1:41691/ using document root <none> and password file <none>
I20260809 09:28:26.587550 26140 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:26.587605 26140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:26.587809 26140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:26.589565 26140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "4a43614c72bc478fbe8eedf316c84dbf"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "b610cb42d93271e6b83e7d08cf39c34f"
server_key_iv: "32483e1dfbd2fc512e259d4f073bd79e"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.590140 26140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "4a43614c72bc478fbe8eedf316c84dbf"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "b610cb42d93271e6b83e7d08cf39c34f"
server_key_iv: "32483e1dfbd2fc512e259d4f073bd79e"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.593362 26140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:28:26.595460 26166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.596572 26140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:28:26.596697 26140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "4a43614c72bc478fbe8eedf316c84dbf"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "b610cb42d93271e6b83e7d08cf39c34f"
server_key_iv: "32483e1dfbd2fc512e259d4f073bd79e"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.596905 26140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:26.600965 26140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:26.601471 26140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:26.601635 26140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:26.602106 26140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:26.602900 26140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:26.602941 26140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.603008 26140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:26.603037 26140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.610832 26140 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:38901
I20260809 09:28:26.610853 26340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:38901 every 8 connection(s)
I20260809 09:28:26.611529 26140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:28:26.616401 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:26.613069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:51873 (local address 127.9.102.62:35097)
0809 09:28:26.613185 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:26.613188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:26.614030 (+ 842us) server_negotiation.cc:407] Connection header received
0809 09:28:26.615040 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:26.615042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:26.615076 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:26.615142 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:26.615999 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:26.616002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:26.616004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:28:26.616014 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:26.616032 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:28:26.616039 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:26.616144 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:26.616266 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 09:28:26.616300 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 09:28:26.617054 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:26.613373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:35097 (local address 127.9.102.1:51873)
0809 09:28:26.613895 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:26.613926 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:26.614877 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:26.615290 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:26.615295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:26.615364 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:26.615785 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:28:26.615786 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:26.615818 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:28:26.615873 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:26.616154 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:26.616161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:26.616165 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 09:28:26.616279 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 09:28:26.616427 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":96,"threads_started":1}
I20260809 09:28:26.618122 26341 heartbeater.cc:347] Connected to a master server at 127.9.102.62:35097
I20260809 09:28:26.618330 26341 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:26.618666 26341 heartbeater.cc:511] Master 127.9.102.62:35097 requested a full tablet report, sending...
I20260809 09:28:26.619935 26049 ts_manager.cc:194] Registered new tserver with Master: 4a43614c72bc478fbe8eedf316c84dbf (127.9.102.1:38901)
I20260809 09:28:26.620573 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 26140
I20260809 09:28:26.620683 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:28:26.621040 9624 external_mini_cluster.cc:1428] Setting key 9c3ae168f3185bcc92145722e513e965
I20260809 09:28:26.622797 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:35097
--builtin_ntp_servers=127.9.102.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 09:28:26.718956 26346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:26.719164 26346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:26.719197 26346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:26.722887 26346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:26.722929 26346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:26.722973 26346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:28:26.726878 26346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:26.727639 26346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:26.728700 26346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:26.736835 26358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:26.736940 26346 server_base.cc:1034] running on GCE node
W20260809 09:28:26.736812 26355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:26.736830 26363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:26.738827 26346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:26.739300 26346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:26.740473 26346 hybrid_clock.cc:648] HybridClock initialized: now 1786267706740448 us; error 34 us; skew 500 ppm
I20260809 09:28:26.742018 26346 webserver.cc:460] Webserver started at http://127.9.102.2:40043/ using document root <none> and password file <none>
I20260809 09:28:26.742483 26346 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:26.742571 26346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:26.742769 26346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:26.744436 26346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "e7e283a87e5e4f558d378268cb6f9bd5"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "a44b3f52c63f12c0bb15cdde7e8136f4"
server_key_iv: "6bd00bb849e53e13b4dc94388f1a0ea1"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.745048 26346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "e7e283a87e5e4f558d378268cb6f9bd5"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "a44b3f52c63f12c0bb15cdde7e8136f4"
server_key_iv: "6bd00bb849e53e13b4dc94388f1a0ea1"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.748297 26346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:28:26.750411 26373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.751436 26346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:28:26.751574 26346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "e7e283a87e5e4f558d378268cb6f9bd5"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "a44b3f52c63f12c0bb15cdde7e8136f4"
server_key_iv: "6bd00bb849e53e13b4dc94388f1a0ea1"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.751675 26346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:26.755656 26346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:26.756264 26346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:26.756438 26346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:26.756937 26346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:26.757773 26346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:26.757841 26346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.757905 26346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:26.757953 26346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.766141 26346 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:46595
I20260809 09:28:26.766166 26547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:46595 every 8 connection(s)
I20260809 09:28:26.766855 26346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:28:26.768957 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 26346
I20260809 09:28:26.769086 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:28:26.769371 9624 external_mini_cluster.cc:1428] Setting key 8e611578ec1538ea913fe7f454ab1cde
I20260809 09:28:26.771126 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:35097
--builtin_ntp_servers=127.9.102.20:37655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 09:28:26.772848 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:26.768493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:40725 (local address 127.9.102.62:35097)
0809 09:28:26.768627 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:26.768630 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:26.769703 (+ 1073us) server_negotiation.cc:407] Connection header received
0809 09:28:26.770970 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:26.770972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:26.771011 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:26.771095 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:26.772382 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:26.772385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:26.772388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:28:26.772400 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:26.772428 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:28:26.772436 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:26.772540 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:26.772693 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 09:28:26.772735 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 09:28:26.773610 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:26.768850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:35097 (local address 127.9.102.2:40725)
0809 09:28:26.769580 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:26.769614 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:26.770794 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:26.771278 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:26.771294 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:26.771460 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:26.772068 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:28:26.772070 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:26.772158 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:28:26.772223 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:26.772530 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:26.772538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:26.772544 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 09:28:26.772719 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 09:28:26.772944 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":148,"threads_started":1}
I20260809 09:28:26.774710 26548 heartbeater.cc:347] Connected to a master server at 127.9.102.62:35097
I20260809 09:28:26.774924 26548 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:26.775285 26548 heartbeater.cc:511] Master 127.9.102.62:35097 requested a full tablet report, sending...
I20260809 09:28:26.776294 26049 ts_manager.cc:194] Registered new tserver with Master: e7e283a87e5e4f558d378268cb6f9bd5 (127.9.102.2:46595)
W20260809 09:28:26.872602 26552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:26.872784 26552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:26.872814 26552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:26.876065 26552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:26.876140 26552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:26.876206 26552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:28:26.880497 26552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:37655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:35097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:26.881219 26552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:26.882200 26552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:26.890141 26559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:26.890220 26552 server_base.cc:1034] running on GCE node
W20260809 09:28:26.890134 26563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:26.890133 26565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:26.890749 26552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:26.891228 26552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:26.892374 26552 hybrid_clock.cc:648] HybridClock initialized: now 1786267706892363 us; error 19 us; skew 500 ppm
I20260809 09:28:26.893998 26552 webserver.cc:460] Webserver started at http://127.9.102.3:36253/ using document root <none> and password file <none>
I20260809 09:28:26.894603 26552 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:26.894681 26552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:26.894929 26552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:26.896754 26552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "9c95b6743e2b4ce8a0d8e1758cdd5c64"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "fc3f578e65c3c12bdbf5def9ddbe3a7b"
server_key_iv: "c766d0bcbf6fa49ce7e953a21580a012"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.897238 26552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "9c95b6743e2b4ce8a0d8e1758cdd5c64"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "fc3f578e65c3c12bdbf5def9ddbe3a7b"
server_key_iv: "c766d0bcbf6fa49ce7e953a21580a012"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.900419 26552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 09:28:26.902557 26578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.903723 26552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:28:26.903831 26552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "9c95b6743e2b4ce8a0d8e1758cdd5c64"
format_stamp: "Formatted at 2026-08-09 09:28:26 on dist-test-slave-7vjv"
server_key: "fc3f578e65c3c12bdbf5def9ddbe3a7b"
server_key_iv: "c766d0bcbf6fa49ce7e953a21580a012"
server_key_version: "encryptionkey@0"
I20260809 09:28:26.903924 26552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:26.907701 26552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:26.908262 26552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:26.908411 26552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:26.908846 26552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:26.909574 26552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:26.909614 26552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.909672 26552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:26.909703 26552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:26.916895 26552 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:36219
I20260809 09:28:26.916915 26768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:36219 every 8 connection(s)
I20260809 09:28:26.917644 26552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:28:26.917824 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 26552
I20260809 09:28:26.917929 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:28:26.918224 9624 external_mini_cluster.cc:1428] Setting key d6157da44fe9eb01f1dff4d3f7941051
I20260809 09:28:26.923133 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:26.919319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:33675 (local address 127.9.102.62:35097)
0809 09:28:26.919437 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:26.919441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:26.920454 (+ 1013us) server_negotiation.cc:407] Connection header received
0809 09:28:26.921878 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:26.921881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:26.921913 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:26.921974 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:26.922818 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:26.922820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:26.922822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:28:26.922832 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:26.922851 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:28:26.922858 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:26.922923 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:26.923013 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 09:28:26.923044 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 09:28:26.923847 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:26.919671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:35097 (local address 127.9.102.3:33675)
0809 09:28:26.920330 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:26.920381 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:26.921718 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:26.922098 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:26.922103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:26.922171 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:26.922607 (+ 436us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:28:26.922608 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:26.922640 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:28:26.922696 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:26.922925 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:26.922931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:26.922935 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 09:28:26.923046 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 09:28:26.923197 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":95,"threads_started":1}
I20260809 09:28:26.924713 26769 heartbeater.cc:347] Connected to a master server at 127.9.102.62:35097
I20260809 09:28:26.924932 26769 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:26.925262 26769 heartbeater.cc:511] Master 127.9.102.62:35097 requested a full tablet report, sending...
I20260809 09:28:26.926110 26049 ts_manager.cc:194] Registered new tserver with Master: 9c95b6743e2b4ce8a0d8e1758cdd5c64 (127.9.102.3:36219)
I20260809 09:28:26.932937 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 09:28:26.935562 26111 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:28:26.934727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49580 (local address 127.9.102.62:35097)
0809 09:28:26.934843 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:26.934863 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:26.934968 (+ 105us) server_negotiation.cc:407] Connection header received
0809 09:28:26.935106 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:26.935108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:26.935134 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:26.935193 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:26.935446 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49580: BlockingRecv error: recv got EOF from 127.0.0.1:49580 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 09:28:26.935688 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 26140
I20260809 09:28:26.955386 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 26346
I20260809 09:28:26.963338 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 26552
I20260809 09:28:26.971853 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 26001
2026-08-09T09:28:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (738 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:28:27 dist-test-slave-7vjv krb5kdc[26785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:28:27 dist-test-slave-7vjv krb5kdc[26785](info): set up 2 sockets
Aug 09 09:28:27 dist-test-slave-7vjv krb5kdc[26785](info): commencing operation
krb5kdc: starting...
W20260809 09:28:28.133138 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:28:28 dist-test-slave-7vjv krb5kdc[26785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:28:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:28:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:28:28.240643 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:38723
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:38723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:28.340581 26804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:28.340883 26804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:28.341037 26804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:28.348363 26804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:28:28.348415 26804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:28.348433 26804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:28.348448 26804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:28:28.348500 26804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:28:28.355002 26804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:38723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:38723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:28.356245 26804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:28.357630 26804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:28.367959 26817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:28.367972 26813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:28.368171 26804 server_base.cc:1034] running on GCE node
W20260809 09:28:28.367974 26812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:28.368922 26804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:28.370172 26804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:28.371356 26804 hybrid_clock.cc:648] HybridClock initialized: now 1786267708371326 us; error 44 us; skew 500 ppm
Aug 09 09:28:28 dist-test-slave-7vjv krb5kdc[26785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:28.373911 26804 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:28:28.374635 26804 webserver.cc:460] Webserver started at http://127.9.102.62:39917/ using document root <none> and password file <none>
I20260809 09:28:28.375154 26804 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:28.375216 26804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:28.375367 26804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:28.377147 26804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "2318869c67824f689196b5a0b39cf7fa"
format_stamp: "Formatted at 2026-08-09 09:28:28 on dist-test-slave-7vjv"
server_key: "bd8586f72e7b25897b41c32a27c82e51"
server_key_iv: "2d7b8bfb058a989593c31a537605145a"
server_key_version: "encryptionkey@0"
I20260809 09:28:28.377606 26804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "2318869c67824f689196b5a0b39cf7fa"
format_stamp: "Formatted at 2026-08-09 09:28:28 on dist-test-slave-7vjv"
server_key: "bd8586f72e7b25897b41c32a27c82e51"
server_key_iv: "2d7b8bfb058a989593c31a537605145a"
server_key_version: "encryptionkey@0"
I20260809 09:28:28.380810 26804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:28:28.382903 26832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:28.384009 26804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:28:28.384174 26804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "2318869c67824f689196b5a0b39cf7fa"
format_stamp: "Formatted at 2026-08-09 09:28:28 on dist-test-slave-7vjv"
server_key: "bd8586f72e7b25897b41c32a27c82e51"
server_key_iv: "2d7b8bfb058a989593c31a537605145a"
server_key_version: "encryptionkey@0"
I20260809 09:28:28.384256 26804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:28.398591 26804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:28.401101 26804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:28.401266 26804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:28.407619 26804 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:38723
I20260809 09:28:28.407864 26921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:38723 every 8 connection(s)
I20260809 09:28:28.408525 26804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:28:28.410921 26923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:28.415654 26923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa: Bootstrap starting.
I20260809 09:28:28.415751 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 26804
I20260809 09:28:28.415903 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:28:28.416200 9624 external_mini_cluster.cc:1428] Setting key 97afacdd04510fa3516be9000de2047b
I20260809 09:28:28.418071 26923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:28.418725 26923 log.cc:824] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa: Log is configured to *not* fsync() on all Append() calls
I20260809 09:28:28.420418 26923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa: No bootstrap required, opened a new log
I20260809 09:28:28.423552 26923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2318869c67824f689196b5a0b39cf7fa" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38723 } }
Aug 09 09:28:28 dist-test-slave-7vjv krb5kdc[26785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:28:28.423756 26923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:28.423836 26923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2318869c67824f689196b5a0b39cf7fa, State: Initialized, Role: FOLLOWER
I20260809 09:28:28.424264 26923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2318869c67824f689196b5a0b39cf7fa" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38723 } }
I20260809 09:28:28.424407 26923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:28:28.424484 26923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:28:28.424571 26923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:28.425504 26923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2318869c67824f689196b5a0b39cf7fa" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38723 } }
I20260809 09:28:28.425824 26923 leader_election.cc:302] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2318869c67824f689196b5a0b39cf7fa; no voters:
I20260809 09:28:28.426133 26923 leader_election.cc:288] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:28:28.426189 26931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:28:28.426379 26931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [term 1 LEADER]: Becoming Leader. State: Replica: 2318869c67824f689196b5a0b39cf7fa, State: Running, Role: LEADER
I20260809 09:28:28.426646 26931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2318869c67824f689196b5a0b39cf7fa" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38723 } }
I20260809 09:28:28.427232 26923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:28:28.427697 26933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2318869c67824f689196b5a0b39cf7fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2318869c67824f689196b5a0b39cf7fa" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38723 } } }
I20260809 09:28:28.427814 26933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:28.428162 26932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2318869c67824f689196b5a0b39cf7fa. Latest consensus state: current_term: 1 leader_uuid: "2318869c67824f689196b5a0b39cf7fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2318869c67824f689196b5a0b39cf7fa" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38723 } } }
I20260809 09:28:28.428269 26932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa [sys.catalog]: This master's current role is: LEADER
I20260809 09:28:28.429124 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:28.417329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.9.102.62:38723)
0809 09:28:28.417713 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:28.417730 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:28.417772 (+ 42us) server_negotiation.cc:407] Connection header received
0809 09:28:28.418364 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:28.418381 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:28.418658 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:28.418942 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:28.419699 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:28.420664 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:28.421434 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:28.421843 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:28.422005 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:28.424350 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:28.424354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:28.424365 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:28.424407 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:28.426014 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:28.426542 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:28.426548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:28.426554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:28.426683 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:28.426984 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:28.426987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:28.426989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:28.427372 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:28.427539 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:28.427988 (+ 449us) server_negotiation.cc:299] Negotiation successful
0809 09:28:28.428288 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":135,"threads_started":1}
W20260809 09:28:28.431562 26954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:28:28.431643 26954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:28:28.431710 26936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:28:28.432444 26936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:28:28.437731 26936 catalog_manager.cc:1347] Generated new cluster ID: a01136a2bd204cc992caae82b495603c
I20260809 09:28:28.437789 26936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:28:28.450105 26936 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:28:28.450891 26936 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:28:28.457607 26936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa: Generated new TSK 0
I20260809 09:28:28.458115 26936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:28:28.487295 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38723
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:28.587805 26965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:28.588007 26965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:28.588078 26965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:28.591552 26965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:28.591600 26965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:28.591692 26965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:28:28.596319 26965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:28.597127 26965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:28.598006 26965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:28.606492 26981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:28.606599 26965 server_base.cc:1034] running on GCE node
W20260809 09:28:28.606523 26976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:28.606496 26970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:28.607164 26965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:28.607685 26965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:28.608834 26965 hybrid_clock.cc:648] HybridClock initialized: now 1786267708608812 us; error 34 us; skew 500 ppm
Aug 09 09:28:28 dist-test-slave-7vjv krb5kdc[26785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:28.611367 26965 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:28:28.612082 26965 webserver.cc:460] Webserver started at http://127.9.102.1:42877/ using document root <none> and password file <none>
I20260809 09:28:28.612622 26965 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:28.612674 26965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:28.612838 26965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:28.614547 26965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "6ebee152ecf14483aaa227db4e812cbc"
format_stamp: "Formatted at 2026-08-09 09:28:28 on dist-test-slave-7vjv"
server_key: "bb47fd6e07d96472a09be687c93edeae"
server_key_iv: "d70bb822ca0958f6c98f5984b70afc98"
server_key_version: "encryptionkey@0"
I20260809 09:28:28.614995 26965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "6ebee152ecf14483aaa227db4e812cbc"
format_stamp: "Formatted at 2026-08-09 09:28:28 on dist-test-slave-7vjv"
server_key: "bb47fd6e07d96472a09be687c93edeae"
server_key_iv: "d70bb822ca0958f6c98f5984b70afc98"
server_key_version: "encryptionkey@0"
I20260809 09:28:28.618181 26965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260809 09:28:28.620247 26991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:28.621342 26965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 09:28:28.621474 26965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "6ebee152ecf14483aaa227db4e812cbc"
format_stamp: "Formatted at 2026-08-09 09:28:28 on dist-test-slave-7vjv"
server_key: "bb47fd6e07d96472a09be687c93edeae"
server_key_iv: "d70bb822ca0958f6c98f5984b70afc98"
server_key_version: "encryptionkey@0"
I20260809 09:28:28.621587 26965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:28.654595 26965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:28.657625 26965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:28.657805 26965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:28.658298 26965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:28.659060 26965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:28.659139 26965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:28.659209 26965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:28.659247 26965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:28.667086 26965 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:41143
I20260809 09:28:28.667127 27181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:41143 every 8 connection(s)
I20260809 09:28:28.667852 26965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:28:28.673142 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 26965
I20260809 09:28:28.673257 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:28:28.673524 9624 external_mini_cluster.cc:1428] Setting key 916dd7442df34e588ab1ccade314f484
Aug 09 09:28:28 dist-test-slave-7vjv krb5kdc[26785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:28:28.679906 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:28.669361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:55879 (local address 127.9.102.62:38723)
0809 09:28:28.669482 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:28.669485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:28.670141 (+ 656us) server_negotiation.cc:407] Connection header received
0809 09:28:28.671194 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:28.671196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:28.671239 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:28.671306 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:28.672443 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:28.672943 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:28.674222 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:28.674419 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:28.674489 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:28.676803 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:28.676806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:28.676807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:28.676827 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:28.678089 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:28.678558 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:28.678563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:28.678569 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:28.678637 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:28.678950 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:28.678953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:28.678954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:28.679112 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:28.679207 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:28.679627 (+ 420us) server_negotiation.cc:299] Negotiation successful
0809 09:28:28.679755 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 09:28:28.680573 27188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:28.669650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38723 (local address 127.9.102.1:55879)
0809 09:28:28.670009 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:28.670046 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:28.671020 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:28.671492 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:28.671501 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:28.671817 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:28.672310 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:28.672329 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:28.673063 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:28.673066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:28.674099 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:28.674108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:28.674563 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:28.674568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:28.674690 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:28.675198 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:28.675220 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:28.676657 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:28.678203 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:28.678208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:28.678220 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:28.678452 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:28.678723 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:28.678727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:28.678729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:28.678861 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:28.679207 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:28.679213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:28.679408 (+ 195us) client_negotiation.cc:769] Sending connection context
0809 09:28:28.679568 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 09:28:28.679852 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
I20260809 09:28:28.681651 27183 heartbeater.cc:347] Connected to a master server at 127.9.102.62:38723
I20260809 09:28:28.681867 27183 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:28.682351 27183 heartbeater.cc:511] Master 127.9.102.62:38723 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:28:28.683656 26861 ts_manager.cc:194] Registered new tserver with Master: 6ebee152ecf14483aaa227db4e812cbc (127.9.102.1:41143)
I20260809 09:28:28.684741 26861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:55879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:28:28.698557 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38723
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:28.795346 27196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:28.795552 27196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:28.795588 27196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:28.799018 27196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:28.799063 27196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:28.799129 27196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:28:28.803700 27196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:28.804528 27196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:28.805366 27196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:28.814404 27211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:28.814410 27207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:28.814504 27196 server_base.cc:1034] running on GCE node
W20260809 09:28:28.814404 27205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:28.815056 27196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:28.815582 27196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:28.816735 27196 hybrid_clock.cc:648] HybridClock initialized: now 1786267708816717 us; error 28 us; skew 500 ppm
Aug 09 09:28:28 dist-test-slave-7vjv krb5kdc[26785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:28.819291 27196 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:28:28.820174 27196 webserver.cc:460] Webserver started at http://127.9.102.2:37889/ using document root <none> and password file <none>
I20260809 09:28:28.820708 27196 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:28.820801 27196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:28.821053 27196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:28.822832 27196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a"
format_stamp: "Formatted at 2026-08-09 09:28:28 on dist-test-slave-7vjv"
server_key: "e210a330513de267e7790fc17654b166"
server_key_iv: "dad22ca04f3cc8de63d5dc4b3b37a233"
server_key_version: "encryptionkey@0"
I20260809 09:28:28.823412 27196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a"
format_stamp: "Formatted at 2026-08-09 09:28:28 on dist-test-slave-7vjv"
server_key: "e210a330513de267e7790fc17654b166"
server_key_iv: "dad22ca04f3cc8de63d5dc4b3b37a233"
server_key_version: "encryptionkey@0"
I20260809 09:28:28.826826 27196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 09:28:28.829288 27220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:28.830366 27196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:28:28.830494 27196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a"
format_stamp: "Formatted at 2026-08-09 09:28:28 on dist-test-slave-7vjv"
server_key: "e210a330513de267e7790fc17654b166"
server_key_iv: "dad22ca04f3cc8de63d5dc4b3b37a233"
server_key_version: "encryptionkey@0"
I20260809 09:28:28.830595 27196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:28.840139 27196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:28.842569 27196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:28.842759 27196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:28.843272 27196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:28.843971 27196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:28.844043 27196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:28.844144 27196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:28.844187 27196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:28.852316 27196 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:34479
I20260809 09:28:28.852366 27409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:34479 every 8 connection(s)
I20260809 09:28:28.853052 27196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:28:28.853608 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 27196
I20260809 09:28:28.853732 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:28:28.853984 9624 external_mini_cluster.cc:1428] Setting key c83a891a7b17c84dcd5325eb5c7e9b4c
Aug 09 09:28:28 dist-test-slave-7vjv krb5kdc[26785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
I20260809 09:28:28.866966 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:28.854737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:36301 (local address 127.9.102.62:38723)
0809 09:28:28.854868 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:28.854872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:28.856436 (+ 1564us) server_negotiation.cc:407] Connection header received
0809 09:28:28.858007 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:28.858009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:28.858060 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:28.858159 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:28.859264 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:28.859801 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:28.860950 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:28.861150 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:28.861253 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:28.863387 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:28.863397 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:28.863399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:28.863434 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:28.864861 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:28.865370 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:28.865373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:28.865374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:28.865484 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:28.865852 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:28.865856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:28.865858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:28.866088 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:28.866200 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:28.866680 (+ 480us) server_negotiation.cc:299] Negotiation successful
0809 09:28:28.866792 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 09:28:28.867628 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:28.854959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38723 (local address 127.9.102.2:36301)
0809 09:28:28.856288 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:28.856343 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:28.857817 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:28.858332 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:28.858340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:28.858695 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:28.859119 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:28.859131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:28.859927 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:28.859930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:28.860828 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:28.860838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:28.861287 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:28.861291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:28.861443 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:28.861936 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:28.861950 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:28.863217 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:28.864975 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:28.864981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:28.864993 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:28.865235 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:28.865607 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:28.865610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:28.865612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:28.865749 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:28.866219 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:28.866226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:28.866434 (+ 208us) client_negotiation.cc:769] Sending connection context
0809 09:28:28.866638 (+ 204us) client_negotiation.cc:240] Negotiation successful
0809 09:28:28.866918 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":145,"threads_started":1}
I20260809 09:28:28.868963 27411 heartbeater.cc:347] Connected to a master server at 127.9.102.62:38723
I20260809 09:28:28.869182 27411 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:28.869612 27411 heartbeater.cc:511] Master 127.9.102.62:38723 requested a full tablet report, sending...
I20260809 09:28:28.870628 26861 ts_manager.cc:194] Registered new tserver with Master: 7e7047fc7d4c49dfa2eaed5f889d3c6a (127.9.102.2:34479)
I20260809 09:28:28.871170 26861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:36301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:28:28.880470 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38723
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:28:28.979292 27417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:28:28.979506 27417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:28:28.979548 27417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:28:28.983021 27417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:28:28.983064 27417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:28:28.983129 27417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:28:28.987851 27417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:28:28.988726 27417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:28:28.989560 27417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:28:28.998149 27436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:28.998211 27417 server_base.cc:1034] running on GCE node
W20260809 09:28:28.998129 27439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:28:28.998162 27437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:28:28.998718 27417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:28:28.999284 27417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:28:29.000437 27417 hybrid_clock.cc:648] HybridClock initialized: now 1786267709000422 us; error 25 us; skew 500 ppm
Aug 09 09:28:28 dist-test-slave-7vjv krb5kdc[26785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:28:29.002946 27417 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:28:29.003710 27417 webserver.cc:460] Webserver started at http://127.9.102.3:45905/ using document root <none> and password file <none>
I20260809 09:28:29.004264 27417 fs_manager.cc:359] Metadata directory not provided
I20260809 09:28:29.004325 27417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:28:29.004541 27417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:28:29.006443 27417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "6e5390cc5ada4b178dc69739b0c88fde"
format_stamp: "Formatted at 2026-08-09 09:28:29 on dist-test-slave-7vjv"
server_key: "b65da2305761f930622fbe3edb7ad178"
server_key_iv: "975ddcb9bb53f324bc365398e55486c5"
server_key_version: "encryptionkey@0"
I20260809 09:28:29.007009 27417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "6e5390cc5ada4b178dc69739b0c88fde"
format_stamp: "Formatted at 2026-08-09 09:28:29 on dist-test-slave-7vjv"
server_key: "b65da2305761f930622fbe3edb7ad178"
server_key_iv: "975ddcb9bb53f324bc365398e55486c5"
server_key_version: "encryptionkey@0"
I20260809 09:28:29.010493 27417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260809 09:28:29.012759 27450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:29.013819 27417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260809 09:28:29.013947 27417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "6e5390cc5ada4b178dc69739b0c88fde"
format_stamp: "Formatted at 2026-08-09 09:28:29 on dist-test-slave-7vjv"
server_key: "b65da2305761f930622fbe3edb7ad178"
server_key_iv: "975ddcb9bb53f324bc365398e55486c5"
server_key_version: "encryptionkey@0"
I20260809 09:28:29.014083 27417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:28:29.030262 27417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:28:29.032760 27417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:28:29.032953 27417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:28:29.033454 27417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:28:29.034191 27417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:28:29.034253 27417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:29.034307 27417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:28:29.034351 27417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:28:29.042284 27417 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:45061
I20260809 09:28:29.042306 27628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:45061 every 8 connection(s)
I20260809 09:28:29.042990 27417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:28:29.045703 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 27417
I20260809 09:28:29.045833 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:28:29.046133 9624 external_mini_cluster.cc:1428] Setting key 9c77881a7d4bd31a48059414f150fb52
Aug 09 09:28:29 dist-test-slave-7vjv krb5kdc[26785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267708, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:28:29.055238 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.044498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:39163 (local address 127.9.102.62:38723)
0809 09:28:29.044633 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:29.044637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:29.045350 (+ 713us) server_negotiation.cc:407] Connection header received
0809 09:28:29.046808 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:29.046811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:29.046862 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:29.046937 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:29.048267 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.048729 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.049705 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.049888 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.049957 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.051947 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:29.051951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:29.051953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:29.051975 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:29.053347 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:29.053827 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:29.053831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:29.053832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:29.053915 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:29.054236 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:29.054240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:29.054242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:29.054437 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:29.054545 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:29.054965 (+ 420us) server_negotiation.cc:299] Negotiation successful
0809 09:28:29.055084 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 09:28:29.055853 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.044840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38723 (local address 127.9.102.3:39163)
0809 09:28:29.045218 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:29.045253 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:29.046605 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:29.047121 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:29.047130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:29.047554 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:28:29.048133 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:29.048152 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.048819 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.048821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:29.049607 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:29.049615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.049998 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.050001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:29.050139 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.050548 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:28:29.050562 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:28:29.051804 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:28:29.053463 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:29.053469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:29.053481 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:29.053718 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:29.054003 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:28:29.054006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:28:29.054008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:28:29.054142 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:28:29.054538 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:28:29.054544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:28:29.054755 (+ 211us) client_negotiation.cc:769] Sending connection context
0809 09:28:29.054923 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 09:28:29.055200 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":117,"threads_started":1}
I20260809 09:28:29.056778 27631 heartbeater.cc:347] Connected to a master server at 127.9.102.62:38723
I20260809 09:28:29.056982 27631 heartbeater.cc:464] Registering TS with master...
I20260809 09:28:29.057376 27631 heartbeater.cc:511] Master 127.9.102.62:38723 requested a full tablet report, sending...
I20260809 09:28:29.058270 26861 ts_manager.cc:194] Registered new tserver with Master: 6e5390cc5ada4b178dc69739b0c88fde (127.9.102.3:45061)
I20260809 09:28:29.058899 26861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:39163
I20260809 09:28:29.059531 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:28:29.067152 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.061249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.9.102.62:38723)
0809 09:28:29.061357 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:29.061361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:29.061510 (+ 149us) server_negotiation.cc:407] Connection header received
0809 09:28:29.061595 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:29.061596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:29.061624 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:29.061678 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:28:29.062211 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.062599 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.063264 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.063427 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.063490 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.064547 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:28:29.064549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:28:29.064551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:28:29.064569 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:28:29.065928 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:29.066254 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:29.066256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:29.066257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:29.066324 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:28:29.066533 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:28:29.066535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:28:29.066535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:28:29.066704 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:28:29.066793 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:29.066927 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 09:28:29.066997 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 09:28:29.069626 26857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:28:29.072166 26857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:28:29.084311 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.079667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.9.102.3:45061)
0809 09:28:29.080061 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:29.080086 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:29.080118 (+ 32us) server_negotiation.cc:407] Connection header received
0809 09:28:29.080185 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:29.080189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:29.080352 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:29.080822 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:29.080934 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.081937 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.083177 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.083695 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.083806 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.083923 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:29.084076 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 09:28:29.084173 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":102,"threads_started":1}
I20260809 09:28:29.084300 27656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.079141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:45061 (local address 127.0.0.1:55240)
0809 09:28:29.079706 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:29.079720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:29.079810 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:29.080502 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:29.080507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:29.080543 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:29.080679 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:29.080702 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.082063 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.082066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:29.083058 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:29.083068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.083808 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.083811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:29.083901 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.083931 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 09:28:29.084048 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 09:28:29.084165 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":38,"thread_start_us":94,"threads_started":1}
I20260809 09:28:29.084555 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.079043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:41143 (local address 127.0.0.1:35980)
0809 09:28:29.079582 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:29.079649 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:29.079759 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:29.080831 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:29.080833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:29.080848 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:29.080930 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:29.080937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.081873 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.081879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:29.083243 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:29.083251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.084192 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.084196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:29.084288 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.084305 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 09:28:29.084351 (+ 46us) client_negotiation.cc:240] Negotiation successful
0809 09:28:29.084418 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":36,"thread_start_us":67,"threads_started":1}
I20260809 09:28:29.084757 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.079375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34479 (local address 127.0.0.1:49690)
0809 09:28:29.079960 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:29.079972 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:29.080064 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:29.080674 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:29.080677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:29.080710 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:29.080829 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:29.080835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.082100 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.082103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:29.082913 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:29.082924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.083991 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.083994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:29.084117 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.084152 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 09:28:29.084227 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 09:28:29.084288 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":19,"thread_start_us":76,"threads_started":1}
I20260809 09:28:29.085148 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.079755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.9.102.2:34479)
0809 09:28:29.080273 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:29.080282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:29.080301 (+ 19us) server_negotiation.cc:407] Connection header received
0809 09:28:29.080364 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:29.080368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:29.080512 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:29.080718 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:29.080921 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.081939 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.083025 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.083725 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.083818 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.083932 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:29.084762 (+ 830us) server_negotiation.cc:299] Negotiation successful
0809 09:28:29.085001 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":119,"threads_started":1}
I20260809 09:28:29.085381 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.079179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.9.102.1:41143)
0809 09:28:29.080357 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:29.080368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:29.080391 (+ 23us) server_negotiation.cc:407] Connection header received
0809 09:28:29.080465 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:29.080471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:29.080674 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:29.081058 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:29.081166 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.081738 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.083331 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.084043 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.084816 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.085073 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:29.085122 (+ 49us) server_negotiation.cc:299] Negotiation successful
0809 09:28:29.085224 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1082,"thread_start_us":104,"threads_started":1}
I20260809 09:28:29.087144 27514 tablet_service.cc:1467] Processing CreateTablet for tablet fec92056690c4bdb817f65c4549da3eb (DEFAULT_TABLE table=test-table [id=b8a9df01df5e44c9bafe2b760a9c626a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:29.087522 27070 tablet_service.cc:1467] Processing CreateTablet for tablet fec92056690c4bdb817f65c4549da3eb (DEFAULT_TABLE table=test-table [id=b8a9df01df5e44c9bafe2b760a9c626a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:29.087520 27306 tablet_service.cc:1467] Processing CreateTablet for tablet fec92056690c4bdb817f65c4549da3eb (DEFAULT_TABLE table=test-table [id=b8a9df01df5e44c9bafe2b760a9c626a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:28:29.088130 27514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec92056690c4bdb817f65c4549da3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:29.088438 27306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec92056690c4bdb817f65c4549da3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:29.088439 27070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec92056690c4bdb817f65c4549da3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:28:29.093506 27665 tablet_bootstrap.cc:492] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde: Bootstrap starting.
I20260809 09:28:29.095225 27667 tablet_bootstrap.cc:492] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a: Bootstrap starting.
I20260809 09:28:29.095737 27665 tablet_bootstrap.cc:654] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:29.096385 27666 tablet_bootstrap.cc:492] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc: Bootstrap starting.
I20260809 09:28:29.096890 27665 log.cc:824] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde: Log is configured to *not* fsync() on all Append() calls
I20260809 09:28:29.097434 27667 tablet_bootstrap.cc:654] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:29.098181 27667 log.cc:824] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a: Log is configured to *not* fsync() on all Append() calls
I20260809 09:28:29.098619 27665 tablet_bootstrap.cc:492] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde: No bootstrap required, opened a new log
I20260809 09:28:29.098769 27666 tablet_bootstrap.cc:654] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc: Neither blocks nor log segments found. Creating new log.
I20260809 09:28:29.098809 27665 ts_tablet_manager.cc:1397] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260809 09:28:29.099471 27666 log.cc:824] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc: Log is configured to *not* fsync() on all Append() calls
I20260809 09:28:29.099987 27667 tablet_bootstrap.cc:492] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a: No bootstrap required, opened a new log
I20260809 09:28:29.100186 27667 ts_tablet_manager.cc:1397] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:28:29.101174 27666 tablet_bootstrap.cc:492] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc: No bootstrap required, opened a new log
I20260809 09:28:29.101379 27666 ts_tablet_manager.cc:1397] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:28:29.101845 27665 raft_consensus.cc:348] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebee152ecf14483aaa227db4e812cbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 41143 } } peers { permanent_uuid: "6e5390cc5ada4b178dc69739b0c88fde" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45061 } } peers { permanent_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34479 } }
I20260809 09:28:29.102109 27665 raft_consensus.cc:374] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:29.102160 27665 raft_consensus.cc:729] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5390cc5ada4b178dc69739b0c88fde, State: Initialized, Role: FOLLOWER
I20260809 09:28:29.102573 27665 consensus_queue.cc:260] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebee152ecf14483aaa227db4e812cbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 41143 } } peers { permanent_uuid: "6e5390cc5ada4b178dc69739b0c88fde" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45061 } } peers { permanent_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34479 } }
I20260809 09:28:29.103355 27665 ts_tablet_manager.cc:1428] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde: Time spent starting tablet: real 0.004s user 0.004s sys 0.002s
I20260809 09:28:29.103417 27667 raft_consensus.cc:348] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebee152ecf14483aaa227db4e812cbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 41143 } } peers { permanent_uuid: "6e5390cc5ada4b178dc69739b0c88fde" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45061 } } peers { permanent_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34479 } }
I20260809 09:28:29.103632 27667 raft_consensus.cc:374] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:29.103664 27631 heartbeater.cc:503] Master 127.9.102.62:38723 was elected leader, sending a full tablet report...
I20260809 09:28:29.103675 27667 raft_consensus.cc:729] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7047fc7d4c49dfa2eaed5f889d3c6a, State: Initialized, Role: FOLLOWER
I20260809 09:28:29.104023 27667 consensus_queue.cc:260] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebee152ecf14483aaa227db4e812cbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 41143 } } peers { permanent_uuid: "6e5390cc5ada4b178dc69739b0c88fde" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45061 } } peers { permanent_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34479 } }
I20260809 09:28:29.104627 27411 heartbeater.cc:503] Master 127.9.102.62:38723 was elected leader, sending a full tablet report...
I20260809 09:28:29.104728 27667 ts_tablet_manager.cc:1428] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260809 09:28:29.104947 27666 raft_consensus.cc:348] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebee152ecf14483aaa227db4e812cbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 41143 } } peers { permanent_uuid: "6e5390cc5ada4b178dc69739b0c88fde" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45061 } } peers { permanent_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34479 } }
I20260809 09:28:29.105126 27666 raft_consensus.cc:374] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:28:29.105161 27666 raft_consensus.cc:729] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ebee152ecf14483aaa227db4e812cbc, State: Initialized, Role: FOLLOWER
I20260809 09:28:29.105662 27666 consensus_queue.cc:260] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebee152ecf14483aaa227db4e812cbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 41143 } } peers { permanent_uuid: "6e5390cc5ada4b178dc69739b0c88fde" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45061 } } peers { permanent_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34479 } }
I20260809 09:28:29.106206 27183 heartbeater.cc:503] Master 127.9.102.62:38723 was elected leader, sending a full tablet report...
I20260809 09:28:29.107246 27666 ts_tablet_manager.cc:1428] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
W20260809 09:28:29.168881 27184 tablet.cc:2369] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:28:29.181022 27671 raft_consensus.cc:482] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:28:29.181217 27671 raft_consensus.cc:504] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebee152ecf14483aaa227db4e812cbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 41143 } } peers { permanent_uuid: "6e5390cc5ada4b178dc69739b0c88fde" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45061 } } peers { permanent_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34479 } }
I20260809 09:28:29.182327 27671 leader_election.cc:288] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ebee152ecf14483aaa227db4e812cbc (127.9.102.1:41143), 7e7047fc7d4c49dfa2eaed5f889d3c6a (127.9.102.2:34479)
I20260809 09:28:29.186508 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.183129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:51127 (local address 127.9.102.2:34479)
0809 09:28:29.183262 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:29.183268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:29.183284 (+ 16us) server_negotiation.cc:407] Connection header received
0809 09:28:29.183330 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:29.183333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:29.183384 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:29.183483 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:29.183807 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.184510 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.185609 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.185965 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.186072 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.186127 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:29.186338 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 09:28:29.186410 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 09:28:29.186504 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.182659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34479 (local address 127.9.102.3:51127)
0809 09:28:29.183004 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:29.183019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:29.183114 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:29.183495 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:29.183499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:29.183521 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:29.183662 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:29.183672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.184619 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.184622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:29.185485 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:29.185494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.186078 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.186081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:29.186177 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.186249 (+ 72us) client_negotiation.cc:769] Sending connection context
0809 09:28:29.186303 (+ 54us) client_negotiation.cc:240] Negotiation successful
0809 09:28:29.186347 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":89,"thread_start_us":101,"threads_started":1}
I20260809 09:28:29.186556 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.182491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:41143 (local address 127.9.102.3:53853)
0809 09:28:29.182610 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:29.182623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:28:29.182694 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:28:29.183854 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:28:29.183857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:28:29.183875 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:28:29.183973 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:29.183983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.184723 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.184725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:29.185486 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:28:29.185495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.186110 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.186113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:28:29.186277 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.186316 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 09:28:29.186387 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 09:28:29.186451 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":74}
I20260809 09:28:29.186587 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.183328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:53853 (local address 127.9.102.1:41143)
0809 09:28:29.183614 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:29.183618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:29.183638 (+ 20us) server_negotiation.cc:407] Connection header received
0809 09:28:29.183713 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:29.183716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:29.183764 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:29.183837 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:28:29.184085 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.184612 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.185613 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.185972 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.186095 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.186152 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:29.186419 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 09:28:29.186481 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260809 09:28:29.187158 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec92056690c4bdb817f65c4549da3eb" candidate_uuid: "6e5390cc5ada4b178dc69739b0c88fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ebee152ecf14483aaa227db4e812cbc" is_pre_election: true
I20260809 09:28:29.187250 27339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec92056690c4bdb817f65c4549da3eb" candidate_uuid: "6e5390cc5ada4b178dc69739b0c88fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a" is_pre_election: true
I20260809 09:28:29.187408 27103 raft_consensus.cc:2454] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e5390cc5ada4b178dc69739b0c88fde in term 0.
I20260809 09:28:29.187536 27339 raft_consensus.cc:2454] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e5390cc5ada4b178dc69739b0c88fde in term 0.
I20260809 09:28:29.187867 27452 leader_election.cc:302] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5390cc5ada4b178dc69739b0c88fde, 6ebee152ecf14483aaa227db4e812cbc; no voters:
I20260809 09:28:29.188148 27671 raft_consensus.cc:2781] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:28:29.188230 27671 raft_consensus.cc:482] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:28:29.188278 27671 raft_consensus.cc:3037] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:29.189116 27671 raft_consensus.cc:504] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebee152ecf14483aaa227db4e812cbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 41143 } } peers { permanent_uuid: "6e5390cc5ada4b178dc69739b0c88fde" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45061 } } peers { permanent_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34479 } }
I20260809 09:28:29.189440 27671 leader_election.cc:288] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [CANDIDATE]: Term 1 election: Requested vote from peers 6ebee152ecf14483aaa227db4e812cbc (127.9.102.1:41143), 7e7047fc7d4c49dfa2eaed5f889d3c6a (127.9.102.2:34479)
I20260809 09:28:29.189740 27103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec92056690c4bdb817f65c4549da3eb" candidate_uuid: "6e5390cc5ada4b178dc69739b0c88fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ebee152ecf14483aaa227db4e812cbc"
I20260809 09:28:29.189740 27339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec92056690c4bdb817f65c4549da3eb" candidate_uuid: "6e5390cc5ada4b178dc69739b0c88fde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a"
I20260809 09:28:29.189870 27103 raft_consensus.cc:3037] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:29.189870 27339 raft_consensus.cc:3037] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:28:29.191083 27339 raft_consensus.cc:2454] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e5390cc5ada4b178dc69739b0c88fde in term 1.
I20260809 09:28:29.191083 27103 raft_consensus.cc:2454] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e5390cc5ada4b178dc69739b0c88fde in term 1.
I20260809 09:28:29.191340 27452 leader_election.cc:302] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5390cc5ada4b178dc69739b0c88fde, 6ebee152ecf14483aaa227db4e812cbc; no voters:
I20260809 09:28:29.191506 27671 raft_consensus.cc:2781] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:28:29.191732 27671 raft_consensus.cc:686] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 1 LEADER]: Becoming Leader. State: Replica: 6e5390cc5ada4b178dc69739b0c88fde, State: Running, Role: LEADER
I20260809 09:28:29.191996 27671 consensus_queue.cc:237] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebee152ecf14483aaa227db4e812cbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 41143 } } peers { permanent_uuid: "6e5390cc5ada4b178dc69739b0c88fde" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45061 } } peers { permanent_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34479 } }
I20260809 09:28:29.194798 26855 catalog_manager.cc:5495] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e5390cc5ada4b178dc69739b0c88fde (127.9.102.3). New cstate: current_term: 1 leader_uuid: "6e5390cc5ada4b178dc69739b0c88fde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ebee152ecf14483aaa227db4e812cbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 41143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e5390cc5ada4b178dc69739b0c88fde" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34479 } health_report { overall_health: UNKNOWN } } }
I20260809 09:28:29.232996 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.229721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.9.102.3:45061)
0809 09:28:29.229847 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 09:28:29.229853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:28:29.229866 (+ 13us) server_negotiation.cc:407] Connection header received
0809 09:28:29.230041 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:28:29.230044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:28:29.230092 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:28:29.230194 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:28:29.230436 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.230872 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.231675 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:28:29.231870 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:28:29.231973 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:28:29.232285 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:28:29.232628 (+ 343us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:28:29.232751 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 09:28:29.232809 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 09:28:29.232856 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 09:28:29.238219 27103 raft_consensus.cc:1261] T fec92056690c4bdb817f65c4549da3eb P 6ebee152ecf14483aaa227db4e812cbc [term 1 FOLLOWER]: Refusing update from remote peer 6e5390cc5ada4b178dc69739b0c88fde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:29.238220 27339 raft_consensus.cc:1261] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a [term 1 FOLLOWER]: Refusing update from remote peer 6e5390cc5ada4b178dc69739b0c88fde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:28:29.238919 27671 consensus_queue.cc:1035] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7047fc7d4c49dfa2eaed5f889d3c6a" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:29.239135 27681 consensus_queue.cc:1035] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebee152ecf14483aaa227db4e812cbc" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 41143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:28:29.246707 27686 mvcc.cc:205] Tried to move back new op lower bound from 7316552537033302016 to 7316552536852717568. Current Snapshot: MvccSnapshot[applied={T|T < 7316552537033302016}]
I20260809 09:28:29.252324 27685 mvcc.cc:205] Tried to move back new op lower bound from 7316552537033302016 to 7316552536852717568. Current Snapshot: MvccSnapshot[applied={T|T < 7316552537033302016 or (T in {7316552537033302016})}]
I20260809 09:28:29.253504 26855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:28:29.253631 26855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:28:29.255738 26855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2318869c67824f689196b5a0b39cf7fa: Sending DeleteTablet for 3 replicas of tablet fec92056690c4bdb817f65c4549da3eb
I20260809 09:28:29.256469 27070 tablet_service.cc:1514] Processing DeleteTablet for tablet fec92056690c4bdb817f65c4549da3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:29 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:35980
I20260809 09:28:29.256595 27306 tablet_service.cc:1514] Processing DeleteTablet for tablet fec92056690c4bdb817f65c4549da3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:29 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:49690
I20260809 09:28:29.256913 27514 tablet_service.cc:1514] Processing DeleteTablet for tablet fec92056690c4bdb817f65c4549da3eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:28:29 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:55240
I20260809 09:28:29.257164 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 26965
I20260809 09:28:29.257392 27700 tablet_replica.cc:330] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde: stopping tablet replica
I20260809 09:28:29.257853 27700 raft_consensus.cc:2229] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:28:29.258221 27700 raft_consensus.cc:2258] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:29.263038 27700 ts_tablet_manager.cc:1905] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:29.264304 27699 tablet_replica.cc:330] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a: stopping tablet replica
I20260809 09:28:29.264591 27699 raft_consensus.cc:2229] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:28:29.264859 27699 raft_consensus.cc:2258] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:28:29.266898 27699 ts_tablet_manager.cc:1905] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:28:29.267165 27700 ts_tablet_manager.cc:1918] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:28:29.267279 27700 log.cc:1196] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/fec92056690c4bdb817f65c4549da3eb
I20260809 09:28:29.267729 27700 ts_tablet_manager.cc:1939] T fec92056690c4bdb817f65c4549da3eb P 6e5390cc5ada4b178dc69739b0c88fde: Deleting consensus metadata
I20260809 09:28:29.269167 26836 catalog_manager.cc:4841] TS 6e5390cc5ada4b178dc69739b0c88fde (127.9.102.3:45061): tablet fec92056690c4bdb817f65c4549da3eb (table test-table [id=b8a9df01df5e44c9bafe2b760a9c626a]) successfully deleted
I20260809 09:28:29.269259 27699 ts_tablet_manager.cc:1918] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:28:29.269337 27699 log.cc:1196] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/fec92056690c4bdb817f65c4549da3eb
I20260809 09:28:29.269626 27699 ts_tablet_manager.cc:1939] T fec92056690c4bdb817f65c4549da3eb P 7e7047fc7d4c49dfa2eaed5f889d3c6a: Deleting consensus metadata
I20260809 09:28:29.270648 26833 catalog_manager.cc:4841] TS 7e7047fc7d4c49dfa2eaed5f889d3c6a (127.9.102.2:34479): tablet fec92056690c4bdb817f65c4549da3eb (table test-table [id=b8a9df01df5e44c9bafe2b760a9c626a]) successfully deleted
W20260809 09:28:29.279848 26834 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:41143 (error 108)
I20260809 09:28:29.280067 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 27196
I20260809 09:28:29.282768 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:28:29.282385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:41143 (local address 127.0.0.1:35996)
0809 09:28:29.282562 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 09:28:29.282652 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:41143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260809 09:28:29.282986 26834 catalog_manager.cc:4584] TS 6ebee152ecf14483aaa227db4e812cbc (127.9.102.1:41143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fec92056690c4bdb817f65c4549da3eb: Network error: Client connection negotiation failed: client connection to 127.9.102.1:41143: connect: Connection refused (error 111)
I20260809 09:28:29.289456 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 27417
I20260809 09:28:29.299119 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 26804
2026-08-09T09:28:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2329 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:28:29 dist-test-slave-7vjv krb5kdc[27707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:28:29 dist-test-slave-7vjv krb5kdc[27707](info): set up 2 sockets
Aug 09 09:28:29 dist-test-slave-7vjv krb5kdc[27707](info): commencing operation
krb5kdc: starting...
W20260809 09:28:30.445734 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:28:30 dist-test-slave-7vjv krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:28:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:28:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/rangeradmin_127.9.102.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/rangeradmin_127.9.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/rangerlookup_127.9.102.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/rangerlookup_127.9.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/HTTP_127.9.102.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/HTTP_127.9.102.20@KRBTEST.COM.keytab.
I20260809 09:28:30.591872 9624 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taska20N0I/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/postgres -l logfile start
2026-08-09 09:28:32.753 UTC [27764] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 09:28:32.753 UTC [27764] LOG: listening on IPv4 address "127.9.102.20", port 46425
2026-08-09 09:28:32.756 UTC [27764] LOG: listening on Unix socket "/tmp/.s.PGSQL.46425"
2026-08-09 09:28:32.787 UTC [27768] LOG: database system was shut down at 2026-08-09 09:28:31 UTC
2026-08-09 09:28:32.791 UTC [27764] LOG: database system is ready to accept connections
I20260809 09:28:33.852192 9624 mini_postgres.cc:96] Postgres bound to 46425
2026-08-09 09:28:33.859 UTC [27788] FATAL: database "slave" does not exist
127.9.102.20:46425 - accepting connections
I20260809 09:28:33.860246 9624 mini_ranger.cc:161] Starting Ranger...
I20260809 09:28:33.874272 9624 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 09:28:34.286525 9624 mini_ranger.cc:87] Created ranger Postgres database
I20260809 09:28:34.286625 9624 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-admin
2026-08-09 09:28:34,567 [I] DB FLAVOR :POSTGRES
2026-08-09 09:28:34,568 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 09:28:34,568 [I] Checking connection..
2026-08-09 09:28:34,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 09:28:34,816 [I] Checking connection passed.
2026-08-09 09:28:34,816 [I] --------- Verifying version history table ---------
2026-08-09 09:28:34,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 09:28:35,059 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 09:28:35,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 09:28:35,287 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 09:28:35,287 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 09:28:35,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 09:28:35.499 UTC [27891] WARNING: there is no transaction in progress
2026-08-09 09:28:35,505 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 09:28:35,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 09:28:35,740 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 09:28:35,740 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 09:28:35,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 09:28:35,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 09:28:36,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7vjv.c.gcp-upstream.internal','N') ;"
2026-08-09 09:28:36,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 09:28:36,663 [I] Table x_portal_user does not exist in database ranger
2026-08-09 09:28:36,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 09:28:36,900 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 09:28:36,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 09:28:37,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 09:28:37,322 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 09:28:37,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 09:28:38.159 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.172 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.187 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.200 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.214 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.229 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.275 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.281 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.293 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.302 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.311 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.323 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.332 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.337 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.343 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38.349 UTC [28075] WARNING: there is no transaction in progress
2026-08-09 09:28:38,886 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 09:28:38,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 09:28:39,132 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 09:28:39,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 09:28:39,382 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 09:28:39,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 09:28:39,618 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 09:28:39,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 09:28:39,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 09:28:40,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7vjv.c.gcp-upstream.internal';"
2026-08-09 09:28:40,285 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 09:28:40,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 09:28:40,495 [I] DB_PATCHES have already been applied
I20260809 09:28:40.501727 9624 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-admin:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taska20N0I/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 09:28:40.501824 9624 mini_ranger.cc:193] Using host: 127.9.102.20
I20260809 09:28:40.505964 9624 mini_ranger.cc:227] Ranger admin URL: http://127.9.102.20:34119
Aug 09, 2026 9:28:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 9:28:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 9:28:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 9:28:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 9:28:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 9:28:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 9:28:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 09:28:41.743595 9624 mini_ranger.cc:160] Time spent starting Ranger: real 7.883s user 0.003s sys 0.008s
Aug 09, 2026 9:29:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 9:29:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 09, 2026 9:29:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 9:29:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 9:29:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 9:29:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 09:29:02.287906 9624 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/rangerkms_127.9.102.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/rangerkms_127.9.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.102.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.102.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.102.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/HTTP_127.9.102.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.102.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/HTTP_127.9.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 09:29:02 dist-test-slave-7vjv krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267742, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 09:29:02.375051 9624 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 09:29:02.385684 9624 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 09:29:03.135848 9624 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 09:29:03.135947 9624 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-kms
I20260809 09:29:03.136045 9624 mini_ranger_kms.cc:227] Using postgres at 127.9.102.20:46425
2026-08-09 09:29:03,595 [I] DB FLAVOR :POSTGRES
2026-08-09 09:29:03,595 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 09:29:03,595 [I] Checking connection
2026-08-09 09:29:03,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 09:29:03,884 [I] connection success
2026-08-09 09:29:03,885 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 09:29:03,885 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 09:29:03,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 09:29:04,136 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 09:29:04,136 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 09:29:04,136 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 09:29:04,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46425/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 09:29:04,406 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 09:29:04.836694 9624 mini_ranger_kms.cc:310] Created kms service
I20260809 09:29:04.995386 9624 mini_ranger_kms.cc:326] Created kudu user
I20260809 09:29:05.040297 9624 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 09:29:05 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267742, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:05.417199 9624 mini_ranger_kms.cc:384] Added ranger policy
I20260809 09:29:05.417341 9624 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-kms:/tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 09:29:05.417378 9624 mini_ranger_kms.cc:240] Using host: 127.9.102.20
I20260809 09:29:05.421856 9624 mini_ranger_kms.cc:276] Ranger KMS PID: 28379
I20260809 09:29:05.421983 9624 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.102.20:41665
Aug 09, 2026 9:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 9:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 9:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 9:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 9:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 9:29:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 09:29:07 dist-test-slave-7vjv krb5kdc[27707](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786267747, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.102.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:07.928920 9624 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.554s user 0.027s sys 0.397s
I20260809 09:29:07.929047 9624 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 09:29:07.929158 9624 mini_ranger_kms.cc:401] 127.9.102.20:41665/kms/v1/keys
Aug 09 09:29:08 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267747, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.102.20@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
Aug 09 09:29:08 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267747, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.102.20@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
Aug 09 09:29:10 dist-test-slave-7vjv krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:29:10.814849 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:46607
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:46607
--ranger_config_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.102.20:41665/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:29:10.932061 28465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:29:10.932343 28465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:29:10.932379 28465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:29:10.935941 28465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:29:10.935993 28465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:29:10.936014 28465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:29:10.936033 28465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:29:10.936074 28465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:29:10.941306 28465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.102.20:41665/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:46607
--ranger_config_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:46607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:29:10.942189 28465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:29:10.943054 28465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:29:10.951684 28474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:10.951666 28471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:10.951916 28465 server_base.cc:1034] running on GCE node
W20260809 09:29:10.951968 28477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:10.952495 28465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:29:10.953445 28465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:29:10.954631 28465 hybrid_clock.cc:648] HybridClock initialized: now 1786267750954612 us; error 35 us; skew 500 ppm
Aug 09 09:29:10 dist-test-slave-7vjv krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267750, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:10.961141 28465 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:29:10.962045 28465 webserver.cc:460] Webserver started at http://127.9.102.62:43859/ using document root <none> and password file <none>
I20260809 09:29:10.962530 28465 fs_manager.cc:359] Metadata directory not provided
I20260809 09:29:10.962604 28465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:29:10.962821 28465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 09:29:10 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267750, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:11.020582 28465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "e513378242904379b84247ace73fbbbb"
format_stamp: "Formatted at 2026-08-09 09:29:11 on dist-test-slave-7vjv"
server_key: "35073ea9b67debd62eacbc22c7d5ebe0"
server_key_iv: "786fd0b1d677017d2bfdcead65c2c843"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:11.021261 28465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "e513378242904379b84247ace73fbbbb"
format_stamp: "Formatted at 2026-08-09 09:29:11 on dist-test-slave-7vjv"
server_key: "35073ea9b67debd62eacbc22c7d5ebe0"
server_key_iv: "786fd0b1d677017d2bfdcead65c2c843"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:11.025123 28465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260809 09:29:11.041592 28486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:11.042908 28465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 09:29:11.043009 28465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "e513378242904379b84247ace73fbbbb"
format_stamp: "Formatted at 2026-08-09 09:29:11 on dist-test-slave-7vjv"
server_key: "35073ea9b67debd62eacbc22c7d5ebe0"
server_key_iv: "786fd0b1d677017d2bfdcead65c2c843"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:11.043148 28465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:29:11.061118 28465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:29:11.063843 28465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:29:11.064069 28465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:29:11.074965 28465 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:46607
I20260809 09:29:11.075064 28588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:46607 every 8 connection(s)
I20260809 09:29:11.075927 28465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:29:11.078692 28591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:29:11.081614 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 28465
I20260809 09:29:11.081737 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:29:11.082010 9624 external_mini_cluster.cc:1428] Setting key 1f2d14839c57c1fc04869608edffc1ca
I20260809 09:29:11.085600 28591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb: Bootstrap starting.
I20260809 09:29:11.088552 28591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb: Neither blocks nor log segments found. Creating new log.
I20260809 09:29:11.089584 28591 log.cc:824] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb: Log is configured to *not* fsync() on all Append() calls
Aug 09 09:29:11 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:29:11.091653 28591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb: No bootstrap required, opened a new log
I20260809 09:29:11.096203 28591 raft_consensus.cc:348] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e513378242904379b84247ace73fbbbb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 46607 } }
I20260809 09:29:11.096459 28591 raft_consensus.cc:374] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:29:11.096560 28591 raft_consensus.cc:729] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e513378242904379b84247ace73fbbbb, State: Initialized, Role: FOLLOWER
I20260809 09:29:11.096889 28600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:11.083311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.9.102.62:46607)
0809 09:29:11.083680 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:11.083697 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:11.083739 (+ 42us) server_negotiation.cc:407] Connection header received
0809 09:29:11.084474 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:11.084493 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:11.084798 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:11.085154 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:11.086038 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:11.087073 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:11.087973 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:11.088558 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:11.088749 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:11.091661 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:11.091672 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:11.091690 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:11.091720 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:11.093853 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:11.094335 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:11.094340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:11.094345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:11.094487 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:11.094776 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:11.094779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:11.094780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:11.095239 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:11.095479 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:11.095718 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 09:29:11.095967 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":131,"threads_started":1}
I20260809 09:29:11.096990 28591 consensus_queue.cc:260] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e513378242904379b84247ace73fbbbb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 46607 } }
I20260809 09:29:11.097149 28591 raft_consensus.cc:388] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:29:11.097220 28591 raft_consensus.cc:482] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:29:11.097313 28591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:29:11.098232 28591 raft_consensus.cc:504] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e513378242904379b84247ace73fbbbb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 46607 } }
I20260809 09:29:11.098610 28591 leader_election.cc:302] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e513378242904379b84247ace73fbbbb; no voters:
I20260809 09:29:11.098940 28591 leader_election.cc:288] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:29:11.099344 28603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:29:11.099530 28603 raft_consensus.cc:686] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [term 1 LEADER]: Becoming Leader. State: Replica: e513378242904379b84247ace73fbbbb, State: Running, Role: LEADER
I20260809 09:29:11.099900 28603 consensus_queue.cc:237] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e513378242904379b84247ace73fbbbb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 46607 } }
I20260809 09:29:11.100306 28591 sys_catalog.cc:564] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:29:11.101176 28603 sys_catalog.cc:455] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e513378242904379b84247ace73fbbbb. Latest consensus state: current_term: 1 leader_uuid: "e513378242904379b84247ace73fbbbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e513378242904379b84247ace73fbbbb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 46607 } } }
I20260809 09:29:11.101387 28603 sys_catalog.cc:458] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [sys.catalog]: This master's current role is: LEADER
I20260809 09:29:11.103237 28604 sys_catalog.cc:455] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e513378242904379b84247ace73fbbbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e513378242904379b84247ace73fbbbb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 46607 } } }
I20260809 09:29:11.103345 28604 sys_catalog.cc:458] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb [sys.catalog]: This master's current role is: LEADER
I20260809 09:29:11.103946 28591 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 09:29:11.104478 28616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:29:11.112232 28616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:29:11.121273 28616 catalog_manager.cc:1347] Generated new cluster ID: 4656b4a608e2456faa7515210d04a3a1
I20260809 09:29:11.121327 28616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:29:11.136574 28616 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:29:11.137903 28616 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:29:11.156246 28616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb: Generated new TSK 0
I20260809 09:29:11.157200 28616 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:29:11.968426 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.102.20:41665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:46607
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:29:12.160442 28675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:29:12.160802 28675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:29:12.160868 28675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:29:12.166407 28675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:29:12.166519 28675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:29:12.166642 28675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:29:12.174544 28675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.102.20:41665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:46607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:29:12.175792 28675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:29:12.177407 28675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:29:12.188874 28682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:12.189155 28683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:12.189337 28686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:12.189397 28675 server_base.cc:1034] running on GCE node
I20260809 09:29:12.189944 28675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:29:12.190752 28675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:29:12.191983 28675 hybrid_clock.cc:648] HybridClock initialized: now 1786267752191951 us; error 36 us; skew 500 ppm
Aug 09 09:29:12 dist-test-slave-7vjv krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:12.195487 28675 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:29:12.196254 28675 webserver.cc:460] Webserver started at http://127.9.102.1:42025/ using document root <none> and password file <none>
I20260809 09:29:12.196765 28675 fs_manager.cc:359] Metadata directory not provided
I20260809 09:29:12.196820 28675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:29:12.197009 28675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 09:29:12 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:12.217438 28675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "fb61a2d3bb9f4946afd6c925ffc68017"
format_stamp: "Formatted at 2026-08-09 09:29:12 on dist-test-slave-7vjv"
server_key: "6b9aaa5eff5a4ea57284ddc9575af6a1"
server_key_iv: "e4af3cbecaec1d40521c93b0192fd395"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:12.218170 28675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "fb61a2d3bb9f4946afd6c925ffc68017"
format_stamp: "Formatted at 2026-08-09 09:29:12 on dist-test-slave-7vjv"
server_key: "6b9aaa5eff5a4ea57284ddc9575af6a1"
server_key_iv: "e4af3cbecaec1d40521c93b0192fd395"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:12.225611 28675 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260809 09:29:12.241058 28702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:12.242937 28675 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 09:29:12.243113 28675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "fb61a2d3bb9f4946afd6c925ffc68017"
format_stamp: "Formatted at 2026-08-09 09:29:12 on dist-test-slave-7vjv"
server_key: "6b9aaa5eff5a4ea57284ddc9575af6a1"
server_key_iv: "e4af3cbecaec1d40521c93b0192fd395"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:12.243269 28675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:29:12.269307 28675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:29:12.272610 28675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:29:12.272789 28675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:29:12.273399 28675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:29:12.274273 28675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:29:12.274345 28675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:12.274464 28675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:29:12.274514 28675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:12.293766 28675 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:35639
I20260809 09:29:12.295218 28675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:29:12.301633 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 28675
I20260809 09:29:12.301776 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:29:12.301977 28887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:35639 every 8 connection(s)
I20260809 09:29:12.302186 9624 external_mini_cluster.cc:1428] Setting key 41b08074d570648f58aef7e37d70dc8b
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
Aug 09 09:29:12 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:29:12.340236 28894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:12.310516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:46607 (local address 127.9.102.1:48567)
0809 09:29:12.313974 (+ 3458us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:12.314030 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:12.315433 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:12.316743 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:12.316752 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:12.317130 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:29:12.317763 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:12.317782 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:12.318901 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:12.318905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:12.320256 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:12.320267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:12.332219 (+ 11952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:12.332222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:12.332399 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:12.333051 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:29:12.333072 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:29:12.334763 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:29:12.337122 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:12.337127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:12.337141 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:12.337440 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:12.337742 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:12.337744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:12.337746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:12.337916 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:12.338395 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:29:12.338401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:29:12.338759 (+ 358us) client_negotiation.cc:769] Sending connection context
0809 09:29:12.338941 (+ 182us) client_negotiation.cc:240] Negotiation successful
0809 09:29:12.339240 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3218,"thread_start_us":264,"threads_started":1}
I20260809 09:29:12.340951 28893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:12.309959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:48567 (local address 127.9.102.62:46607)
0809 09:29:12.316235 (+ 6276us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:12.316239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:12.316254 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:29:12.316295 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:12.316297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:12.316342 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:12.316439 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:12.317953 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:12.318756 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:12.320392 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:12.320736 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:12.320836 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:12.334954 (+ 14118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:12.334974 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:12.334977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:12.335002 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:12.336998 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:12.337547 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:12.337550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:12.337552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:12.337649 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:12.338019 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:12.338022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:12.338023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:12.338237 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:12.338342 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:12.340671 (+ 2329us) server_negotiation.cc:299] Negotiation successful
0809 09:29:12.340793 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6203,"thread_start_us":81,"threads_started":1}
I20260809 09:29:12.344460 28890 heartbeater.cc:347] Connected to a master server at 127.9.102.62:46607
I20260809 09:29:12.344709 28890 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:12.345278 28890 heartbeater.cc:511] Master 127.9.102.62:46607 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
I20260809 09:29:12.347275 28525 ts_manager.cc:194] Registered new tserver with Master: fb61a2d3bb9f4946afd6c925ffc68017 (127.9.102.1:35639)
I20260809 09:29:12.348798 28525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:48567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:29:12.365291 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.102.20:41665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:46607
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 09:29:12 dist-test-slave-7vjv krb5kdc[27707](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786267752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 09:29:12.508994 28903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:29:12.509264 28903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:29:12.509325 28903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:29:12.513671 28903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:29:12.513720 28903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:29:12.513803 28903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:29:12.518519 28903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.102.20:41665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:46607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:29:12.519407 28903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:29:12.520522 28903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:29:12.529796 28915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:12.529932 28911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:12.530339 28920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:12.530581 28903 server_base.cc:1034] running on GCE node
I20260809 09:29:12.530946 28903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:29:12.531522 28903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:29:12.532675 28903 hybrid_clock.cc:648] HybridClock initialized: now 1786267752532655 us; error 33 us; skew 500 ppm
Aug 09 09:29:12 dist-test-slave-7vjv krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:12.535267 28903 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:29:12.536034 28903 webserver.cc:460] Webserver started at http://127.9.102.2:43051/ using document root <none> and password file <none>
I20260809 09:29:12.536566 28903 fs_manager.cc:359] Metadata directory not provided
I20260809 09:29:12.536650 28903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:29:12.536870 28903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 09:29:12 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:12.551169 28903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "9cb8cf37def84e39b9e2f90ca11c53e5"
format_stamp: "Formatted at 2026-08-09 09:29:12 on dist-test-slave-7vjv"
server_key: "5b134c6373616935b6b03339a1ab505d"
server_key_iv: "752ecef7e75a8677c39636ff90ae84b8"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:12.552037 28903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "9cb8cf37def84e39b9e2f90ca11c53e5"
format_stamp: "Formatted at 2026-08-09 09:29:12 on dist-test-slave-7vjv"
server_key: "5b134c6373616935b6b03339a1ab505d"
server_key_iv: "752ecef7e75a8677c39636ff90ae84b8"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:12.556949 28903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260809 09:29:12.570398 28931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:12.571759 28903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260809 09:29:12.571940 28903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "9cb8cf37def84e39b9e2f90ca11c53e5"
format_stamp: "Formatted at 2026-08-09 09:29:12 on dist-test-slave-7vjv"
server_key: "5b134c6373616935b6b03339a1ab505d"
server_key_iv: "752ecef7e75a8677c39636ff90ae84b8"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:12.572150 28903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:29:12.587224 28903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:29:12.589983 28903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:29:12.590173 28903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:29:12.590847 28903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:29:12.591753 28903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:29:12.591833 28903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:12.591964 28903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:29:12.592001 28903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:12.605947 28903 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:34761
I20260809 09:29:12.606011 29139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:34761 every 8 connection(s)
I20260809 09:29:12.606890 28903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:29:12.612615 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 28903
I20260809 09:29:12.612756 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:29:12.613036 9624 external_mini_cluster.cc:1428] Setting key 71396649594b431f9c9a19138b817a77
Aug 09 09:29:12 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:29:12.631286 28893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:12.615480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:39951 (local address 127.9.102.62:46607)
0809 09:29:12.615599 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:12.615603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:12.618494 (+ 2891us) server_negotiation.cc:407] Connection header received
0809 09:29:12.619363 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:12.619366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:12.619413 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:12.619482 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:12.621214 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:12.621948 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:12.623708 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:12.624081 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:12.624193 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:12.627404 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:12.627408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:12.627410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:12.627437 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:12.629015 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:12.629584 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:12.629587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:12.629589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:12.629681 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:12.630061 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:12.630064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:12.630066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:12.630253 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:12.630337 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:12.631031 (+ 694us) server_negotiation.cc:299] Negotiation successful
0809 09:29:12.631143 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:29:12.632773 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:12.614416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:46607 (local address 127.9.102.2:39951)
0809 09:29:12.618355 (+ 3939us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:12.618414 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:12.619192 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:12.619833 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:12.619842 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:12.620306 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:29:12.621070 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:12.621092 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:12.622108 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:12.622114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:12.623573 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:12.623587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:12.624401 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:12.624407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:12.624665 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:12.625414 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:29:12.625432 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:29:12.627221 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:29:12.629146 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:12.629151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:12.629163 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:12.629476 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:12.629790 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:12.629793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:12.629794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:12.629959 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:12.630429 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:29:12.630435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:29:12.630769 (+ 334us) client_negotiation.cc:769] Sending connection context
0809 09:29:12.631471 (+ 702us) client_negotiation.cc:240] Negotiation successful
0809 09:29:12.631749 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3737,"thread_start_us":162,"threads_started":1}
I20260809 09:29:12.633970 29142 heartbeater.cc:347] Connected to a master server at 127.9.102.62:46607
I20260809 09:29:12.634171 29142 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:12.634752 29142 heartbeater.cc:511] Master 127.9.102.62:46607 requested a full tablet report, sending...
I20260809 09:29:12.636080 28525 ts_manager.cc:194] Registered new tserver with Master: 9cb8cf37def84e39b9e2f90ca11c53e5 (127.9.102.2:34761)
I20260809 09:29:12.636946 28525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:39951
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:29:12.673511 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.102.20:41665/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:46607
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:29:12.829623 29152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:29:12.830003 29152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:29:12.830130 29152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:29:12.835410 29152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:29:12.835538 29152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:29:12.835675 29152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:29:12.846313 29152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.102.20:41665/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:46607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:29:12.848179 29152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:29:12.849725 29152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:29:12.869380 29161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:12.869844 29163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:12.870383 29160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:12.871032 29152 server_base.cc:1034] running on GCE node
I20260809 09:29:12.871479 29152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:29:12.872197 29152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:29:12.873387 29152 hybrid_clock.cc:648] HybridClock initialized: now 1786267752873328 us; error 71 us; skew 500 ppm
Aug 09 09:29:12 dist-test-slave-7vjv krb5kdc[27707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:12.876972 29152 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:29:12.877965 29152 webserver.cc:460] Webserver started at http://127.9.102.3:40477/ using document root <none> and password file <none>
I20260809 09:29:12.878883 29152 fs_manager.cc:359] Metadata directory not provided
I20260809 09:29:12.878975 29152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:29:12.879173 29152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 09:29:12 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:12.905042 29152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "8e0ea9ddcad04c47ba2a24fd471adabb"
format_stamp: "Formatted at 2026-08-09 09:29:12 on dist-test-slave-7vjv"
server_key: "777d6fa68fc578abae6b694b783a21fc"
server_key_iv: "30920605951b9af9a311952e1db29854"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:12.905818 29152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "8e0ea9ddcad04c47ba2a24fd471adabb"
format_stamp: "Formatted at 2026-08-09 09:29:12 on dist-test-slave-7vjv"
server_key: "777d6fa68fc578abae6b694b783a21fc"
server_key_iv: "30920605951b9af9a311952e1db29854"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:12.910825 29152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260809 09:29:12.930773 29176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:12.932431 29152 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260809 09:29:12.932571 29152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "8e0ea9ddcad04c47ba2a24fd471adabb"
format_stamp: "Formatted at 2026-08-09 09:29:12 on dist-test-slave-7vjv"
server_key: "777d6fa68fc578abae6b694b783a21fc"
server_key_iv: "30920605951b9af9a311952e1db29854"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:12.932677 29152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:29:12.975073 29152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:29:12.978632 29152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:29:12.978850 29152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:29:12.979522 29152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:29:12.980707 29152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:29:12.980821 29152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:12.980919 29152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:29:12.980978 29152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:13.006481 29152 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:45147
I20260809 09:29:13.007681 29152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:29:13.012051 29375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:45147 every 8 connection(s)
I20260809 09:29:13.017696 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 29152
I20260809 09:29:13.017853 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:29:13.018177 9624 external_mini_cluster.cc:1428] Setting key 5d57458ca5ef52818441436152100bd6
Aug 09 09:29:13 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:29:13.025615 28893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.011703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:54023 (local address 127.9.102.62:46607)
0809 09:29:13.011820 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:13.011824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:13.012715 (+ 891us) server_negotiation.cc:407] Connection header received
0809 09:29:13.014066 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:13.014070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:13.014121 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:13.014168 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:13.015603 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.016573 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.018268 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.018602 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.018675 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.021751 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:13.021755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:13.021757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:13.021782 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:13.023451 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:13.024012 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:13.024015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:13.024016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:13.024118 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:13.024492 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:13.024495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:13.024497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:13.024689 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:13.024754 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:13.025349 (+ 595us) server_negotiation.cc:299] Negotiation successful
0809 09:29:13.025473 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:29:13.027168 29380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.010618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:46607 (local address 127.9.102.3:54023)
0809 09:29:13.012571 (+ 1953us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:13.012624 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:13.013842 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:13.014541 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:13.014549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:13.014860 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:29:13.015447 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:13.015465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.016737 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.016740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:13.018125 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:13.018137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.018901 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.018904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:13.019081 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.019797 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:29:13.019818 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:29:13.021581 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:29:13.023582 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:13.023586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:13.023601 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:13.023900 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:13.024222 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:13.024225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:13.024227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:13.024388 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:13.024828 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:29:13.024834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:29:13.025111 (+ 277us) client_negotiation.cc:769] Sending connection context
0809 09:29:13.025987 (+ 876us) client_negotiation.cc:240] Negotiation successful
0809 09:29:13.026261 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1726,"thread_start_us":207,"threads_started":1}
I20260809 09:29:13.028483 29377 heartbeater.cc:347] Connected to a master server at 127.9.102.62:46607
I20260809 09:29:13.028681 29377 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:13.029152 29377 heartbeater.cc:511] Master 127.9.102.62:46607 requested a full tablet report, sending...
I20260809 09:29:13.030216 28525 ts_manager.cc:194] Registered new tserver with Master: 8e0ea9ddcad04c47ba2a24fd471adabb (127.9.102.3:45147)
I20260809 09:29:13.030893 28525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:54023
I20260809 09:29:13.032244 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:29:13.047804 28893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.036508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.9.102.62:46607)
0809 09:29:13.036595 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:13.036598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:13.040323 (+ 3725us) server_negotiation.cc:407] Connection header received
0809 09:29:13.040501 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:13.040504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:13.040558 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:13.040612 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:13.041588 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.042468 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.043288 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.043501 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.043598 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.044839 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:13.044842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:13.044845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:13.044867 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:13.046178 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:13.046594 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:13.046599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:13.046600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:13.046684 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:13.046933 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:13.046936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:13.046938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:13.047135 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:13.047303 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:13.047551 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 09:29:13.047653 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 09:29:13.051090 28524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:29:13.054431 28524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 09 09:29:13 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:13.095778 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.086968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:45147 (local address 127.0.0.1:43506)
0809 09:29:13.088138 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:13.088184 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:13.088305 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:13.089896 (+ 1591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:13.089913 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:13.089941 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:29:13.090073 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:13.090080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.092304 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.092307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:13.093528 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:13.093539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.095288 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.095291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:13.095404 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.095448 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 09:29:13.095550 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 09:29:13.095655 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"thread_start_us":130,"threads_started":1}
I20260809 09:29:13.096696 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.087069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.9.102.3:45147)
0809 09:29:13.089373 (+ 2304us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:13.089394 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:13.089416 (+ 22us) server_negotiation.cc:407] Connection header received
0809 09:29:13.089490 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:13.089495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:13.089689 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:13.089876 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:29:13.090211 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.092179 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.093746 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.095165 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.096268 (+ 1103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.096418 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:13.096471 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 09:29:13.096549 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2207,"thread_start_us":91,"threads_started":1}
I20260809 09:29:13.099771 29235 tablet_service.cc:1467] Processing CreateTablet for tablet d5b74a2b18304a6b83e6821e4caac9de (DEFAULT_TABLE table=test-table [id=ea3652205ff0421bbcdfa1e152a0c3a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:29:13.100484 29394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.088112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:35639 (local address 127.0.0.1:51328)
0809 09:29:13.092544 (+ 4432us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:13.092555 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:13.092626 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:13.093228 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:13.093231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:13.093247 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:29:13.093353 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:13.093359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.094602 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.094605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:13.095751 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:13.095763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.099902 (+ 4139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.099907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:13.100001 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.100027 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 09:29:13.100105 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 09:29:13.100182 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4385,"thread_start_us":71,"threads_started":1}
I20260809 09:29:13.100936 29235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b74a2b18304a6b83e6821e4caac9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:29:13.104578 29403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.088440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.9.102.1:35639)
0809 09:29:13.088718 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:13.088725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:13.092712 (+ 3987us) server_negotiation.cc:407] Connection header received
0809 09:29:13.092791 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:13.092796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:13.092968 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:13.093098 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:29:13.093477 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.094467 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.097319 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.098062 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.099663 (+ 1601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.099818 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:13.104317 (+ 4499us) server_negotiation.cc:299] Negotiation successful
0809 09:29:13.104432 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260809 09:29:13.106927 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.087699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34761 (local address 127.0.0.1:40166)
0809 09:29:13.092278 (+ 4579us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:13.092291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:13.092386 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:13.097100 (+ 4714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:13.097103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:13.097120 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:29:13.097197 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:13.097203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.101575 (+ 4372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.101578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:13.103634 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:13.103646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.105863 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.105867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:13.105954 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.105976 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 09:29:13.106046 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 09:29:13.106103 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4518,"thread_start_us":96,"threads_started":1}
I20260809 09:29:13.109897 28783 tablet_service.cc:1467] Processing CreateTablet for tablet d5b74a2b18304a6b83e6821e4caac9de (DEFAULT_TABLE table=test-table [id=ea3652205ff0421bbcdfa1e152a0c3a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:29:13.110910 28783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b74a2b18304a6b83e6821e4caac9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:29:13.116793 29404 tablet_bootstrap.cc:492] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb: Bootstrap starting.
I20260809 09:29:13.122152 29404 tablet_bootstrap.cc:654] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb: Neither blocks nor log segments found. Creating new log.
I20260809 09:29:13.123405 29404 log.cc:824] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb: Log is configured to *not* fsync() on all Append() calls
I20260809 09:29:13.125603 29404 tablet_bootstrap.cc:492] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb: No bootstrap required, opened a new log
I20260809 09:29:13.125783 29404 ts_tablet_manager.cc:1397] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 09:29:13.130421 29404 raft_consensus.cc:348] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45147 } } peers { permanent_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34761 } } peers { permanent_uuid: "fb61a2d3bb9f4946afd6c925ffc68017" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35639 } }
I20260809 09:29:13.130658 29404 raft_consensus.cc:374] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:29:13.130714 29404 raft_consensus.cc:729] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0ea9ddcad04c47ba2a24fd471adabb, State: Initialized, Role: FOLLOWER
I20260809 09:29:13.131356 29404 consensus_queue.cc:260] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45147 } } peers { permanent_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34761 } } peers { permanent_uuid: "fb61a2d3bb9f4946afd6c925ffc68017" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35639 } }
I20260809 09:29:13.132441 29404 ts_tablet_manager.cc:1428] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 09:29:13.132792 29377 heartbeater.cc:503] Master 127.9.102.62:46607 was elected leader, sending a full tablet report...
I20260809 09:29:13.134125 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.087879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.9.102.2:34761)
0809 09:29:13.096195 (+ 8316us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:13.096226 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:13.096248 (+ 22us) server_negotiation.cc:407] Connection header received
0809 09:29:13.096320 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:13.096324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:13.096688 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:13.096972 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:29:13.098315 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.101442 (+ 3127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.104823 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.105579 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.105682 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.105801 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:13.133871 (+ 28070us) server_negotiation.cc:299] Negotiation successful
0809 09:29:13.133990 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8165,"thread_start_us":109,"threads_started":1}
I20260809 09:29:13.136727 29022 tablet_service.cc:1467] Processing CreateTablet for tablet d5b74a2b18304a6b83e6821e4caac9de (DEFAULT_TABLE table=test-table [id=ea3652205ff0421bbcdfa1e152a0c3a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:29:13.137717 29022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b74a2b18304a6b83e6821e4caac9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:29:13.144408 29411 tablet_bootstrap.cc:492] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017: Bootstrap starting.
I20260809 09:29:13.146440 29412 tablet_bootstrap.cc:492] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5: Bootstrap starting.
I20260809 09:29:13.149078 29412 tablet_bootstrap.cc:654] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5: Neither blocks nor log segments found. Creating new log.
I20260809 09:29:13.150315 29412 log.cc:824] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5: Log is configured to *not* fsync() on all Append() calls
I20260809 09:29:13.150588 29411 tablet_bootstrap.cc:654] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017: Neither blocks nor log segments found. Creating new log.
I20260809 09:29:13.151539 29411 log.cc:824] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017: Log is configured to *not* fsync() on all Append() calls
I20260809 09:29:13.152875 29412 tablet_bootstrap.cc:492] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5: No bootstrap required, opened a new log
I20260809 09:29:13.153074 29412 ts_tablet_manager.cc:1397] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260809 09:29:13.158955 29412 raft_consensus.cc:348] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45147 } } peers { permanent_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34761 } } peers { permanent_uuid: "fb61a2d3bb9f4946afd6c925ffc68017" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35639 } }
I20260809 09:29:13.159250 29412 raft_consensus.cc:374] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:29:13.159358 29412 raft_consensus.cc:729] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb8cf37def84e39b9e2f90ca11c53e5, State: Initialized, Role: FOLLOWER
I20260809 09:29:13.159952 29412 consensus_queue.cc:260] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45147 } } peers { permanent_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34761 } } peers { permanent_uuid: "fb61a2d3bb9f4946afd6c925ffc68017" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35639 } }
I20260809 09:29:13.161206 29412 ts_tablet_manager.cc:1428] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5: Time spent starting tablet: real 0.008s user 0.001s sys 0.007s
I20260809 09:29:13.161355 29142 heartbeater.cc:503] Master 127.9.102.62:46607 was elected leader, sending a full tablet report...
I20260809 09:29:13.163486 29411 tablet_bootstrap.cc:492] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017: No bootstrap required, opened a new log
I20260809 09:29:13.163683 29411 ts_tablet_manager.cc:1397] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20260809 09:29:13.172420 29411 raft_consensus.cc:348] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45147 } } peers { permanent_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34761 } } peers { permanent_uuid: "fb61a2d3bb9f4946afd6c925ffc68017" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35639 } }
I20260809 09:29:13.172719 29411 raft_consensus.cc:374] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:29:13.172820 29411 raft_consensus.cc:729] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb61a2d3bb9f4946afd6c925ffc68017, State: Initialized, Role: FOLLOWER
I20260809 09:29:13.173363 29411 consensus_queue.cc:260] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45147 } } peers { permanent_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34761 } } peers { permanent_uuid: "fb61a2d3bb9f4946afd6c925ffc68017" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35639 } }
I20260809 09:29:13.174608 29411 ts_tablet_manager.cc:1428] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20260809 09:29:13.175014 28890 heartbeater.cc:503] Master 127.9.102.62:46607 was elected leader, sending a full tablet report...
W20260809 09:29:13.259018 29379 tablet.cc:2369] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:29:13.312784 28891 tablet.cc:2369] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:29:13.363152 29143 tablet.cc:2369] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 09:29:13 dist-test-slave-7vjv krb5kdc[27707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267752, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:13.446974 29406 raft_consensus.cc:482] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:29:13.447247 29406 raft_consensus.cc:504] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45147 } } peers { permanent_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34761 } } peers { permanent_uuid: "fb61a2d3bb9f4946afd6c925ffc68017" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35639 } }
I20260809 09:29:13.448724 29406 leader_election.cc:288] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb8cf37def84e39b9e2f90ca11c53e5 (127.9.102.2:34761), fb61a2d3bb9f4946afd6c925ffc68017 (127.9.102.1:35639)
I20260809 09:29:13.454908 29380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.448996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34761 (local address 127.9.102.3:52069)
0809 09:29:13.449148 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:13.449165 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:13.449266 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:13.450665 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:13.450669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:13.450690 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:29:13.450816 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:13.450827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.451927 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.451930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:13.453282 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:13.453295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.454414 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.454418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:13.454578 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.454604 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 09:29:13.454685 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 09:29:13.454757 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 09:29:13.456408 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.450134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:52069 (local address 127.9.102.2:34761)
0809 09:29:13.450244 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:13.450249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:13.450266 (+ 17us) server_negotiation.cc:407] Connection header received
0809 09:29:13.450330 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:13.450333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:13.450408 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:13.450486 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:29:13.451017 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.451759 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.453500 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.454167 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.454278 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.454348 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:13.456180 (+ 1832us) server_negotiation.cc:299] Negotiation successful
0809 09:29:13.456285 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 09:29:13.457067 29050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b74a2b18304a6b83e6821e4caac9de" candidate_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5" is_pre_election: true
I20260809 09:29:13.457437 29050 raft_consensus.cc:2454] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0ea9ddcad04c47ba2a24fd471adabb in term 0.
I20260809 09:29:13.458480 29177 leader_election.cc:302] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0ea9ddcad04c47ba2a24fd471adabb, 9cb8cf37def84e39b9e2f90ca11c53e5; no voters:
I20260809 09:29:13.458827 29406 raft_consensus.cc:2781] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:29:13.458906 29406 raft_consensus.cc:482] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:29:13.458946 29406 raft_consensus.cc:3037] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:29:13.460832 29406 raft_consensus.cc:504] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45147 } } peers { permanent_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34761 } } peers { permanent_uuid: "fb61a2d3bb9f4946afd6c925ffc68017" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35639 } }
I20260809 09:29:13.461072 29380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.449327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:35639 (local address 127.9.102.3:59383)
0809 09:29:13.455064 (+ 5737us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:13.455079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:13.455158 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:13.456963 (+ 1805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:13.456971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:13.456992 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:29:13.457111 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:13.457119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.458296 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.458299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:13.459376 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:13.459395 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.460538 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.460542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:13.460665 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.460702 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 09:29:13.460792 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 09:29:13.460868 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5685,"thread_start_us":79,"threads_started":1}
I20260809 09:29:13.461218 29406 leader_election.cc:288] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [CANDIDATE]: Term 1 election: Requested vote from peers 9cb8cf37def84e39b9e2f90ca11c53e5 (127.9.102.2:34761), fb61a2d3bb9f4946afd6c925ffc68017 (127.9.102.1:35639)
I20260809 09:29:13.461594 29403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.449872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:59383 (local address 127.9.102.1:35639)
0809 09:29:13.450041 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:13.450047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:13.455690 (+ 5643us) server_negotiation.cc:407] Connection header received
0809 09:29:13.455760 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:13.455764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:13.455814 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:13.455913 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:29:13.457330 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.458041 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.459555 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.460232 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.460352 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.460439 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:13.461395 (+ 956us) server_negotiation.cc:299] Negotiation successful
0809 09:29:13.461472 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 09:29:13.461834 29050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b74a2b18304a6b83e6821e4caac9de" candidate_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5"
I20260809 09:29:13.461956 29050 raft_consensus.cc:3037] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:29:13.462182 28784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b74a2b18304a6b83e6821e4caac9de" candidate_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb61a2d3bb9f4946afd6c925ffc68017" is_pre_election: true
I20260809 09:29:13.462458 28784 raft_consensus.cc:2454] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e0ea9ddcad04c47ba2a24fd471adabb in term 0.
I20260809 09:29:13.462908 28785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b74a2b18304a6b83e6821e4caac9de" candidate_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb61a2d3bb9f4946afd6c925ffc68017"
I20260809 09:29:13.463003 28785 raft_consensus.cc:3037] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:29:13.463594 29050 raft_consensus.cc:2454] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0ea9ddcad04c47ba2a24fd471adabb in term 1.
I20260809 09:29:13.463923 29177 leader_election.cc:302] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e0ea9ddcad04c47ba2a24fd471adabb, 9cb8cf37def84e39b9e2f90ca11c53e5; no voters:
I20260809 09:29:13.464076 29406 raft_consensus.cc:2781] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:29:13.464237 28785 raft_consensus.cc:2454] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e0ea9ddcad04c47ba2a24fd471adabb in term 1.
I20260809 09:29:13.464286 29406 raft_consensus.cc:686] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 1 LEADER]: Becoming Leader. State: Replica: 8e0ea9ddcad04c47ba2a24fd471adabb, State: Running, Role: LEADER
I20260809 09:29:13.464630 29406 consensus_queue.cc:237] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45147 } } peers { permanent_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34761 } } peers { permanent_uuid: "fb61a2d3bb9f4946afd6c925ffc68017" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35639 } }
I20260809 09:29:13.468101 28524 catalog_manager.cc:5495] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e0ea9ddcad04c47ba2a24fd471adabb (127.9.102.3). New cstate: current_term: 1 leader_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e0ea9ddcad04c47ba2a24fd471adabb" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb61a2d3bb9f4946afd6c925ffc68017" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35639 } health_report { overall_health: UNKNOWN } } }
I20260809 09:29:13.555171 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.550438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.9.102.3:45147)
0809 09:29:13.550578 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:13.550583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:13.550813 (+ 230us) server_negotiation.cc:407] Connection header received
0809 09:29:13.551020 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:13.551023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:13.551086 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:13.551188 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:29:13.551450 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.552158 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.553358 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:13.553718 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:13.553789 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:13.554210 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:29:13.554727 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:29:13.554808 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:13.554975 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 09:29:13.555028 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 09:29:13.561767 28785 raft_consensus.cc:1261] T d5b74a2b18304a6b83e6821e4caac9de P fb61a2d3bb9f4946afd6c925ffc68017 [term 1 FOLLOWER]: Refusing update from remote peer 8e0ea9ddcad04c47ba2a24fd471adabb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:29:13.562496 29426 consensus_queue.cc:1035] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb61a2d3bb9f4946afd6c925ffc68017" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:29:13.568565 29050 raft_consensus.cc:1261] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5 [term 1 FOLLOWER]: Refusing update from remote peer 8e0ea9ddcad04c47ba2a24fd471adabb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:29:13.569238 29426 consensus_queue.cc:1035] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cb8cf37def84e39b9e2f90ca11c53e5" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:29:13.580739 29433 mvcc.cc:205] Tried to move back new op lower bound from 7316552718580809728 to 7316552718194442240. Current Snapshot: MvccSnapshot[applied={T|T < 7316552718580809728}]
I20260809 09:29:13.588994 29436 mvcc.cc:205] Tried to move back new op lower bound from 7316552718580809728 to 7316552718194442240. Current Snapshot: MvccSnapshot[applied={T|T < 7316552718580809728}]
I20260809 09:29:13.591336 29434 mvcc.cc:205] Tried to move back new op lower bound from 7316552718580809728 to 7316552718194442240. Current Snapshot: MvccSnapshot[applied={T|T < 7316552718580809728}]
I20260809 09:29:13.599223 28524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:29:13.599355 28524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:29:13.601683 28524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e513378242904379b84247ace73fbbbb: Sending DeleteTablet for 3 replicas of tablet d5b74a2b18304a6b83e6821e4caac9de
I20260809 09:29:13.602490 29235 tablet_service.cc:1514] Processing DeleteTablet for tablet d5b74a2b18304a6b83e6821e4caac9de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:29:13 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:43506
I20260809 09:29:13.602622 29022 tablet_service.cc:1514] Processing DeleteTablet for tablet d5b74a2b18304a6b83e6821e4caac9de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:29:13 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:40166
I20260809 09:29:13.603423 28783 tablet_service.cc:1514] Processing DeleteTablet for tablet d5b74a2b18304a6b83e6821e4caac9de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:29:13 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:51328
I20260809 09:29:13.604724 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 28675
I20260809 09:29:13.613867 29448 tablet_replica.cc:330] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5: stopping tablet replica
I20260809 09:29:13.614234 29448 raft_consensus.cc:2229] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:29:13.614509 29448 raft_consensus.cc:2258] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:29:13.617209 29448 ts_tablet_manager.cc:1905] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:29:13.617477 29447 tablet_replica.cc:330] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb: stopping tablet replica
I20260809 09:29:13.617781 29447 raft_consensus.cc:2229] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:29:13.618171 29447 raft_consensus.cc:2258] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:29:13.619623 29447 ts_tablet_manager.cc:1905] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:29:13.619809 29448 ts_tablet_manager.cc:1918] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:29:13.619906 29448 log.cc:1196] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/d5b74a2b18304a6b83e6821e4caac9de
I20260809 09:29:13.620222 29448 ts_tablet_manager.cc:1939] T d5b74a2b18304a6b83e6821e4caac9de P 9cb8cf37def84e39b9e2f90ca11c53e5: Deleting consensus metadata
I20260809 09:29:13.621290 28488 catalog_manager.cc:4841] TS 9cb8cf37def84e39b9e2f90ca11c53e5 (127.9.102.2:34761): tablet d5b74a2b18304a6b83e6821e4caac9de (table test-table [id=ea3652205ff0421bbcdfa1e152a0c3a5]) successfully deleted
I20260809 09:29:13.622572 29447 ts_tablet_manager.cc:1918] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:29:13.622663 29447 log.cc:1196] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/d5b74a2b18304a6b83e6821e4caac9de
I20260809 09:29:13.622931 29447 ts_tablet_manager.cc:1939] T d5b74a2b18304a6b83e6821e4caac9de P 8e0ea9ddcad04c47ba2a24fd471adabb: Deleting consensus metadata
I20260809 09:29:13.623836 28495 catalog_manager.cc:4841] TS 8e0ea9ddcad04c47ba2a24fd471adabb (127.9.102.3:45147): tablet d5b74a2b18304a6b83e6821e4caac9de (table test-table [id=ea3652205ff0421bbcdfa1e152a0c3a5]) successfully deleted
W20260809 09:29:13.628508 28493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:35639 (error 108)
I20260809 09:29:13.632259 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 28903
I20260809 09:29:13.639281 29452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:13.637770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:35639 (local address 127.0.0.1:51344)
0809 09:29:13.639133 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:13.639198 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1292,"thread_start_us":58,"threads_started":1}
W20260809 09:29:13.644204 28493 catalog_manager.cc:4584] TS fb61a2d3bb9f4946afd6c925ffc68017 (127.9.102.1:35639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5b74a2b18304a6b83e6821e4caac9de: Network error: Client connection negotiation failed: client connection to 127.9.102.1:35639: connect: Connection refused (error 111)
I20260809 09:29:13.649122 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 29152
I20260809 09:29:13.661487 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 28465
I20260809 09:29:13.671821 9624 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 09:29:14.025817 9624 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 09:29:14.025990 9624 mini_ranger.cc:66] Stopping Ranger...
I20260809 09:29:14.371711 9624 mini_ranger.cc:68] Stopped Ranger
2026-08-09 09:29:14.371 UTC [27764] LOG: received smart shutdown request
2026-08-09 09:29:14.375 UTC [27764] LOG: background worker "logical replication launcher" (PID 27784) exited with exit code 1
2026-08-09 09:29:14.376 UTC [27770] LOG: shutting down
2026-08-09 09:29:14.516 UTC [27764] LOG: database system is shut down
2026-08-09T09:29:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45330 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:29:14 dist-test-slave-7vjv krb5kdc[29472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:29:14 dist-test-slave-7vjv krb5kdc[29472](info): set up 2 sockets
Aug 09 09:29:14 dist-test-slave-7vjv krb5kdc[29472](info): commencing operation
krb5kdc: starting...
W20260809 09:29:15.790872 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:29:15 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:29:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/rangeradmin_127.9.102.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/rangeradmin_127.9.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/rangerlookup_127.9.102.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/rangerlookup_127.9.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/HTTP_127.9.102.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/HTTP_127.9.102.20@KRBTEST.COM.keytab.
I20260809 09:29:15.940450 9624 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taska20N0I/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/postgres -l logfile start
2026-08-09 09:29:18.110 UTC [29523] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 09:29:18.110 UTC [29523] LOG: listening on IPv4 address "127.9.102.20", port 46343
2026-08-09 09:29:18.113 UTC [29523] LOG: listening on Unix socket "/tmp/.s.PGSQL.46343"
2026-08-09 09:29:18.133 UTC [29529] LOG: database system was shut down at 2026-08-09 09:29:16 UTC
2026-08-09 09:29:18.136 UTC [29523] LOG: database system is ready to accept connections
I20260809 09:29:19.196997 9624 mini_postgres.cc:96] Postgres bound to 46343
2026-08-09 09:29:19.204 UTC [29544] FATAL: database "slave" does not exist
127.9.102.20:46343 - accepting connections
I20260809 09:29:19.204869 9624 mini_ranger.cc:161] Starting Ranger...
I20260809 09:29:19.219305 9624 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 09:29:19.610406 9624 mini_ranger.cc:87] Created ranger Postgres database
I20260809 09:29:19.610473 9624 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-admin
2026-08-09 09:29:19,873 [I] DB FLAVOR :POSTGRES
2026-08-09 09:29:19,874 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 09:29:19,874 [I] Checking connection..
2026-08-09 09:29:19,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 09:29:20,118 [I] Checking connection passed.
2026-08-09 09:29:20,118 [I] --------- Verifying version history table ---------
2026-08-09 09:29:20,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 09:29:20,369 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 09:29:20,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 09:29:20,617 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 09:29:20,618 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 09:29:20,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 09:29:20.846 UTC [29658] WARNING: there is no transaction in progress
2026-08-09 09:29:20,852 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 09:29:20,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 09:29:21,066 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 09:29:21,066 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 09:29:21,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 09:29:21,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 09:29:21,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7vjv.c.gcp-upstream.internal','N') ;"
2026-08-09 09:29:21,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 09:29:22,018 [I] Table x_portal_user does not exist in database ranger
2026-08-09 09:29:22,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 09:29:22,249 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 09:29:22,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 09:29:22,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 09:29:22,705 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 09:29:22,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 09:29:23.500 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.514 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.528 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.541 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.553 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.568 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.612 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.618 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.630 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.639 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.647 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.659 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.667 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.672 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.678 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:23.684 UTC [29843] WARNING: there is no transaction in progress
2026-08-09 09:29:24,215 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 09:29:24,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 09:29:24,438 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 09:29:24,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 09:29:24,688 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 09:29:24,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 09:29:24,926 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 09:29:24,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 09:29:25,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 09:29:25,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7vjv.c.gcp-upstream.internal';"
2026-08-09 09:29:25,624 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 09:29:25,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 09:29:25,851 [I] DB_PATCHES have already been applied
I20260809 09:29:25.857342 9624 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-admin:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taska20N0I/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 09:29:25.857400 9624 mini_ranger.cc:193] Using host: 127.9.102.20
I20260809 09:29:25.861487 9624 mini_ranger.cc:227] Ranger admin URL: http://127.9.102.20:37925
Aug 09, 2026 9:29:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 9:29:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 9:29:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 9:29:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 9:29:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 9:29:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 9:29:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 09:29:27.190783 9624 mini_ranger.cc:160] Time spent starting Ranger: real 7.986s user 0.003s sys 0.007s
Aug 09, 2026 9:29:44 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 9:29:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 09, 2026 9:29:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 9:29:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 9:29:44 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 9:29:45 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 09:29:45.869669 9624 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/rangerkms_127.9.102.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/rangerkms_127.9.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.102.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.102.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.102.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/HTTP_127.9.102.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.102.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/HTTP_127.9.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 09:29:45 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267785, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 09:29:45.962335 9624 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 09:29:45.972828 9624 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 09:29:46.717674 9624 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 09:29:46.717759 9624 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-kms
I20260809 09:29:46.717779 9624 mini_ranger_kms.cc:227] Using postgres at 127.9.102.20:46343
2026-08-09 09:29:46,877 [I] DB FLAVOR :POSTGRES
2026-08-09 09:29:46,877 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 09:29:46,877 [I] Checking connection
2026-08-09 09:29:46,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 09:29:47,099 [I] connection success
2026-08-09 09:29:47,099 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 09:29:47,099 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 09:29:47,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 09:29:47,319 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 09:29:47,319 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 09:29:47,320 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 09:29:47,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.102.20:46343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 09:29:47,570 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 09:29:47.983980 9624 mini_ranger_kms.cc:310] Created kms service
I20260809 09:29:48.165373 9624 mini_ranger_kms.cc:326] Created kudu user
I20260809 09:29:48.205185 9624 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 09:29:48 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267785, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:48.577244 9624 mini_ranger_kms.cc:384] Added ranger policy
I20260809 09:29:48.577385 9624 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-kms:/tmp/dist-test-taska20N0I/build/debug/bin/postgresql.jar:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taska20N0I/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 09:29:48.577404 9624 mini_ranger_kms.cc:240] Using host: 127.9.102.20
I20260809 09:29:48.581841 9624 mini_ranger_kms.cc:276] Ranger KMS PID: 30150
I20260809 09:29:48.581934 9624 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.102.20:60281
Aug 09, 2026 9:29:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 9:29:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 9:29:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 9:29:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 9:29:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 9:29:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 09:29:50 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786267790, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.102.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 09 09:29:51 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267790, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.102.20@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:52.154883 9624 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.193s user 0.014s sys 0.132s
I20260809 09:29:52.155133 9624 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 09:29:52.155275 9624 mini_ranger_kms.cc:401] 127.9.102.20:60281/kms/v1/keys
Aug 09 09:29:52 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267790, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.102.20@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
Aug 09 09:29:55 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:29:55.190081 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39097
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39097
--ranger_config_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.102.20:60281/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:29:55.296955 30225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:29:55.297168 30225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:29:55.297202 30225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:29:55.300819 30225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:29:55.300866 30225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:29:55.300884 30225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:29:55.300897 30225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:29:55.300910 30225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:29:55.305891 30225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.102.20:60281/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39097
--ranger_config_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:29:55.306722 30225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:29:55.307583 30225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:29:55.315781 30237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:55.315824 30238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:55.315938 30241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:55.316152 30225 server_base.cc:1034] running on GCE node
I20260809 09:29:55.316790 30225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:29:55.318153 30225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:29:55.319371 30225 hybrid_clock.cc:648] HybridClock initialized: now 1786267795319328 us; error 43 us; skew 500 ppm
Aug 09 09:29:55 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267795, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:55.322463 30225 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:29:55.323263 30225 webserver.cc:460] Webserver started at http://127.9.102.62:33307/ using document root <none> and password file <none>
I20260809 09:29:55.323729 30225 fs_manager.cc:359] Metadata directory not provided
I20260809 09:29:55.323800 30225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:29:55.323997 30225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 09:29:55 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267795, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:55.372785 30225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "afcc511c926d422a9cb976b267b80ccb"
format_stamp: "Formatted at 2026-08-09 09:29:55 on dist-test-slave-7vjv"
server_key: "bbb1222b0ff89e011cf3edc95dea1c84"
server_key_iv: "5f2c4f98dc92d1675e15f2bf5428e854"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:55.373647 30225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "afcc511c926d422a9cb976b267b80ccb"
format_stamp: "Formatted at 2026-08-09 09:29:55 on dist-test-slave-7vjv"
server_key: "bbb1222b0ff89e011cf3edc95dea1c84"
server_key_iv: "5f2c4f98dc92d1675e15f2bf5428e854"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:55.378636 30225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 09:29:55.393134 30250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:55.394735 30225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 09:29:55.394867 30225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "afcc511c926d422a9cb976b267b80ccb"
format_stamp: "Formatted at 2026-08-09 09:29:55 on dist-test-slave-7vjv"
server_key: "bbb1222b0ff89e011cf3edc95dea1c84"
server_key_iv: "5f2c4f98dc92d1675e15f2bf5428e854"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:55.394987 30225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:29:55.410862 30225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:29:55.413733 30225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:29:55.413894 30225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:29:55.422489 30335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:39097 every 8 connection(s)
I20260809 09:29:55.422497 30225 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:39097
I20260809 09:29:55.423583 30225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:29:55.426326 30336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:29:55.426644 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 30225
I20260809 09:29:55.426767 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:29:55.427047 9624 external_mini_cluster.cc:1428] Setting key 919b080125d2b42b36d9c7e377c036ae
I20260809 09:29:55.433465 30336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb: Bootstrap starting.
I20260809 09:29:55.436015 30336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb: Neither blocks nor log segments found. Creating new log.
I20260809 09:29:55.436923 30336 log.cc:824] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb: Log is configured to *not* fsync() on all Append() calls
Aug 09 09:29:55 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:29:55.438987 30336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb: No bootstrap required, opened a new log
I20260809 09:29:55.442723 30343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:55.428405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.9.102.62:39097)
0809 09:29:55.428829 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:55.428847 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:55.428925 (+ 78us) server_negotiation.cc:407] Connection header received
0809 09:29:55.429801 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:55.429826 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:55.430237 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:55.430609 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:55.431487 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:55.432937 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:55.434022 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:55.434676 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:55.434838 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:55.437911 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:55.437916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:55.437927 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:55.437951 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:55.440021 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:55.440463 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:55.440474 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:55.440479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:55.440618 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:55.440906 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:55.440916 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:55.440917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:55.441282 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:55.441483 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:55.441777 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 09:29:55.442061 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":118,"threads_started":1}
I20260809 09:29:55.442859 30336 raft_consensus.cc:348] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc511c926d422a9cb976b267b80ccb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39097 } }
I20260809 09:29:55.443090 30336 raft_consensus.cc:374] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:29:55.443176 30336 raft_consensus.cc:729] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afcc511c926d422a9cb976b267b80ccb, State: Initialized, Role: FOLLOWER
I20260809 09:29:55.443599 30336 consensus_queue.cc:260] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc511c926d422a9cb976b267b80ccb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39097 } }
I20260809 09:29:55.443825 30336 raft_consensus.cc:388] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:29:55.443897 30336 raft_consensus.cc:482] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:29:55.444001 30336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:29:55.445070 30336 raft_consensus.cc:504] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc511c926d422a9cb976b267b80ccb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39097 } }
I20260809 09:29:55.445427 30336 leader_election.cc:302] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afcc511c926d422a9cb976b267b80ccb; no voters:
I20260809 09:29:55.445732 30336 leader_election.cc:288] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:29:55.445868 30347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:29:55.446170 30347 raft_consensus.cc:686] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 1 LEADER]: Becoming Leader. State: Replica: afcc511c926d422a9cb976b267b80ccb, State: Running, Role: LEADER
I20260809 09:29:55.446527 30347 consensus_queue.cc:237] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc511c926d422a9cb976b267b80ccb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39097 } }
I20260809 09:29:55.446908 30336 sys_catalog.cc:564] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:29:55.447578 30347 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [sys.catalog]: SysCatalogTable state changed. Reason: New leader afcc511c926d422a9cb976b267b80ccb. Latest consensus state: current_term: 1 leader_uuid: "afcc511c926d422a9cb976b267b80ccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc511c926d422a9cb976b267b80ccb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39097 } } }
I20260809 09:29:55.447687 30347 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [sys.catalog]: This master's current role is: LEADER
I20260809 09:29:55.448335 30349 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afcc511c926d422a9cb976b267b80ccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc511c926d422a9cb976b267b80ccb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39097 } } }
I20260809 09:29:55.448487 30349 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [sys.catalog]: This master's current role is: LEADER
I20260809 09:29:55.448773 30360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:29:55.449060 30336 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 09:29:55.456435 30360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:29:55.463429 30360 catalog_manager.cc:1347] Generated new cluster ID: cbc9cab69bc94c84a4707a1b805c3a59
I20260809 09:29:55.463524 30360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:29:55.500029 30360 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:29:55.501426 30360 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:29:55.512998 30360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb: Generated new TSK 0
I20260809 09:29:55.513784 30360 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:29:56.126593 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.102.20:60281/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39097
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:29:56.261680 30411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:29:56.261973 30411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:29:56.262037 30411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:29:56.266072 30411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:29:56.266151 30411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:29:56.266256 30411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:29:56.270800 30411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.102.20:60281/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:29:56.271672 30411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:29:56.272639 30411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:29:56.281716 30423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:56.281716 30421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:56.282150 30420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:56.282840 30411 server_base.cc:1034] running on GCE node
I20260809 09:29:56.283182 30411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:29:56.283805 30411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:29:56.285035 30411 hybrid_clock.cc:648] HybridClock initialized: now 1786267796285017 us; error 35 us; skew 500 ppm
Aug 09 09:29:56 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:56.287966 30411 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:29:56.288705 30411 webserver.cc:460] Webserver started at http://127.9.102.1:35865/ using document root <none> and password file <none>
I20260809 09:29:56.289202 30411 fs_manager.cc:359] Metadata directory not provided
I20260809 09:29:56.289290 30411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:29:56.289522 30411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 09:29:56 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:56.310171 30411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "85f4ac75f58b483d9aa14f073ff98183"
format_stamp: "Formatted at 2026-08-09 09:29:56 on dist-test-slave-7vjv"
server_key: "b446ac2cbd31a7f1f40f4e1a291e73ca"
server_key_iv: "222d71eb7b3a3a6298271b173d0b2608"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:56.311049 30411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "85f4ac75f58b483d9aa14f073ff98183"
format_stamp: "Formatted at 2026-08-09 09:29:56 on dist-test-slave-7vjv"
server_key: "b446ac2cbd31a7f1f40f4e1a291e73ca"
server_key_iv: "222d71eb7b3a3a6298271b173d0b2608"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:56.321313 30411 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260809 09:29:56.344651 30432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:56.346760 30411 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260809 09:29:56.346885 30411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "85f4ac75f58b483d9aa14f073ff98183"
format_stamp: "Formatted at 2026-08-09 09:29:56 on dist-test-slave-7vjv"
server_key: "b446ac2cbd31a7f1f40f4e1a291e73ca"
server_key_iv: "222d71eb7b3a3a6298271b173d0b2608"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:56.347014 30411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:29:56.372424 30411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:29:56.375803 30411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:29:56.376017 30411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:29:56.376699 30411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:29:56.377652 30411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:29:56.377727 30411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:56.377807 30411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:29:56.377837 30411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:56.388890 30411 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:38551
I20260809 09:29:56.389820 30411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:29:56.390797 30631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:38551 every 8 connection(s)
I20260809 09:29:56.393683 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 30411
I20260809 09:29:56.393796 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:29:56.394121 9624 external_mini_cluster.cc:1428] Setting key 9e6c8606971b8ddbde256430033459e0
Aug 09 09:29:56 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:29:56.406843 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:56.392323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39097 (local address 127.9.102.1:59603)
0809 09:29:56.392869 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:56.392913 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:56.393827 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:56.394799 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:56.394808 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:56.395123 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:29:56.395827 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:56.395858 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:56.396904 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:56.396909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:56.398022 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:56.398032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:56.398756 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:56.398762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:56.398955 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:56.399660 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:29:56.399681 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:29:56.401351 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:29:56.403741 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:56.403748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:56.403761 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:56.404091 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:56.404450 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:56.404455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:56.404457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:56.404631 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:56.405140 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:29:56.405146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:29:56.405479 (+ 333us) client_negotiation.cc:769] Sending connection context
0809 09:29:56.405680 (+ 201us) client_negotiation.cc:240] Negotiation successful
0809 09:29:56.405978 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":131,"threads_started":1}
I20260809 09:29:56.407583 30635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:56.392345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:59603 (local address 127.9.102.62:39097)
0809 09:29:56.392635 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:56.392643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:56.393001 (+ 358us) server_negotiation.cc:407] Connection header received
0809 09:29:56.394070 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:56.394075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:56.394132 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:56.394199 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:56.395999 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:56.396744 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:56.398144 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:56.398498 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:56.398592 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:56.401525 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:56.401530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:56.401536 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:56.401565 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:56.403594 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:56.404229 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:56.404233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:56.404236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:56.404332 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:56.404742 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:56.404746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:56.404748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:56.404970 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:56.405067 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:56.407377 (+ 2310us) server_negotiation.cc:299] Negotiation successful
0809 09:29:56.407473 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":115,"threads_started":1}
I20260809 09:29:56.408524 30632 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
I20260809 09:29:56.408867 30632 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:56.409409 30632 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
I20260809 09:29:56.411595 30273 ts_manager.cc:194] Registered new tserver with Master: 85f4ac75f58b483d9aa14f073ff98183 (127.9.102.1:38551)
I20260809 09:29:56.413079 30273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:59603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:29:56.424746 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.102.20:60281/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39097
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 09:29:56 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786267796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 09:29:56.556317 30639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:29:56.556623 30639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:29:56.556684 30639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:29:56.562302 30639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:29:56.562412 30639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:29:56.562511 30639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:29:56.569238 30639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.102.20:60281/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:29:56.570264 30639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:29:56.571349 30639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:29:56.582422 30654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:56.582597 30649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:56.582660 30648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:56.584225 30639 server_base.cc:1034] running on GCE node
I20260809 09:29:56.584786 30639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:29:56.585498 30639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:29:56.586700 30639 hybrid_clock.cc:648] HybridClock initialized: now 1786267796586681 us; error 35 us; skew 500 ppm
Aug 09 09:29:56 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:56.590274 30639 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:29:56.591415 30639 webserver.cc:460] Webserver started at http://127.9.102.2:45893/ using document root <none> and password file <none>
I20260809 09:29:56.592090 30639 fs_manager.cc:359] Metadata directory not provided
I20260809 09:29:56.592211 30639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:29:56.592511 30639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 09:29:56 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:56.613054 30639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "3a0a244e7a2e4d869e4f2deb6971cf30"
format_stamp: "Formatted at 2026-08-09 09:29:56 on dist-test-slave-7vjv"
server_key: "2dbac31589d9dc57e82c6e7f9935f8cf"
server_key_iv: "cfccb1a477034be9e5dc0db58d8e2138"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:56.615533 30639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "3a0a244e7a2e4d869e4f2deb6971cf30"
format_stamp: "Formatted at 2026-08-09 09:29:56 on dist-test-slave-7vjv"
server_key: "2dbac31589d9dc57e82c6e7f9935f8cf"
server_key_iv: "cfccb1a477034be9e5dc0db58d8e2138"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:56.623715 30639 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260809 09:29:56.650400 30665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:56.661257 30639 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.000s
I20260809 09:29:56.661403 30639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "3a0a244e7a2e4d869e4f2deb6971cf30"
format_stamp: "Formatted at 2026-08-09 09:29:56 on dist-test-slave-7vjv"
server_key: "2dbac31589d9dc57e82c6e7f9935f8cf"
server_key_iv: "cfccb1a477034be9e5dc0db58d8e2138"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:56.661535 30639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:29:56.682003 30639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:29:56.685350 30639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:29:56.685547 30639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:29:56.686286 30639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:29:56.687278 30639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:29:56.687356 30639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:56.687412 30639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:29:56.687486 30639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:56.699963 30639 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:46333
I20260809 09:29:56.701656 30639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:29:56.702785 30842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:46333 every 8 connection(s)
I20260809 09:29:56.704536 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 30639
I20260809 09:29:56.704658 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:29:56.704944 9624 external_mini_cluster.cc:1428] Setting key 0790e93fa3f3f67dc2064455b31fd2e5
Aug 09 09:29:56 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:29:56.721380 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:56.705595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39097 (local address 127.9.102.2:47851)
0809 09:29:56.707679 (+ 2084us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:56.707753 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:56.708948 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:56.709732 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:56.709740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:56.710084 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:29:56.710690 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:56.710711 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:56.711727 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:56.711730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:56.712932 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:56.712944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:56.713683 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:56.713686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:56.713863 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:56.714524 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:29:56.714543 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:29:56.716236 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:29:56.718189 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:56.718195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:56.718208 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:56.718584 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:56.719009 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:56.719013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:56.719016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:56.719205 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:56.719703 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:29:56.719708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:29:56.720006 (+ 298us) client_negotiation.cc:769] Sending connection context
0809 09:29:56.720191 (+ 185us) client_negotiation.cc:240] Negotiation successful
0809 09:29:56.720462 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1864,"thread_start_us":175,"threads_started":1}
I20260809 09:29:56.722033 30635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:56.706951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:47851 (local address 127.9.102.62:39097)
0809 09:29:56.707074 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:56.707079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:56.707903 (+ 824us) server_negotiation.cc:407] Connection header received
0809 09:29:56.709223 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:56.709227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:56.709278 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:56.709331 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:56.710877 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:56.711562 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:56.713068 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:56.713430 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:56.713532 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:56.716403 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:56.716406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:56.716409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:56.716435 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:56.718067 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:56.718734 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:56.718738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:56.718739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:56.718870 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:56.719318 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:56.719321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:56.719323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:56.719566 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:56.719688 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:56.721755 (+ 2067us) server_negotiation.cc:299] Negotiation successful
0809 09:29:56.721882 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:29:56.723394 30845 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39097
I20260809 09:29:56.723585 30845 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:56.724047 30845 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
I20260809 09:29:56.725101 30273 ts_manager.cc:194] Registered new tserver with Master: 3a0a244e7a2e4d869e4f2deb6971cf30 (127.9.102.2:46333)
I20260809 09:29:56.725730 30273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:47851
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:29:56.756649 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.102.20:60281/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39097
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 09:29:56 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
W20260809 09:29:56.905028 30852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:29:56.905414 30852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:29:56.905548 30852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:29:56.910928 30852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:29:56.911047 30852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:29:56.911177 30852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:29:56.917830 30852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.102.20:60281/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:29:56.919137 30852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:29:56.920348 30852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:29:56.937711 30868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:56.938247 30859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:56.939044 30858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:56.939682 30852 server_base.cc:1034] running on GCE node
I20260809 09:29:56.940181 30852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:29:56.940892 30852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:29:56.944172 30852 hybrid_clock.cc:648] HybridClock initialized: now 1786267796944105 us; error 91 us; skew 500 ppm
Aug 09 09:29:56 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:56.947723 30852 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:29:56.948819 30852 webserver.cc:460] Webserver started at http://127.9.102.3:44877/ using document root <none> and password file <none>
I20260809 09:29:56.949508 30852 fs_manager.cc:359] Metadata directory not provided
I20260809 09:29:56.949641 30852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:29:56.949892 30852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 09:29:56 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:56.970548 30852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "a30654a3df214f73ae1b58f560a910af"
format_stamp: "Formatted at 2026-08-09 09:29:56 on dist-test-slave-7vjv"
server_key: "b0358c5ed8c2e2d44b44d0ce8caae673"
server_key_iv: "567523dde30036f90d8fb99bef17bd01"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:56.971596 30852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "a30654a3df214f73ae1b58f560a910af"
format_stamp: "Formatted at 2026-08-09 09:29:56 on dist-test-slave-7vjv"
server_key: "b0358c5ed8c2e2d44b44d0ce8caae673"
server_key_iv: "567523dde30036f90d8fb99bef17bd01"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:56.976397 30852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 09:29:57.000432 30879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:57.002025 30852 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260809 09:29:57.002194 30852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "a30654a3df214f73ae1b58f560a910af"
format_stamp: "Formatted at 2026-08-09 09:29:56 on dist-test-slave-7vjv"
server_key: "b0358c5ed8c2e2d44b44d0ce8caae673"
server_key_iv: "567523dde30036f90d8fb99bef17bd01"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:57.002372 30852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:29:57.026590 30852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:29:57.030089 30852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:29:57.030318 30852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:29:57.031006 30852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:29:57.031970 30852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:29:57.032061 30852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:57.032166 30852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:29:57.032226 30852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:57.045545 30852 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:40951
I20260809 09:29:57.046617 30852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:29:57.046918 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 30852
I20260809 09:29:57.047047 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:29:57.047370 9624 external_mini_cluster.cc:1428] Setting key 9a1fa674f2e8c8fe616efae4a680cc59
I20260809 09:29:57.047962 31064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:40951 every 8 connection(s)
Aug 09 09:29:57 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267796, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:29:57.061424 30635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:57.050604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:57645 (local address 127.9.102.62:39097)
0809 09:29:57.050761 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:57.050765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:57.051330 (+ 565us) server_negotiation.cc:407] Connection header received
0809 09:29:57.052356 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:57.052359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:57.052409 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:57.052481 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:57.053475 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:57.053956 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:57.055084 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:57.055301 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:57.055409 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:57.057758 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:57.057761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:57.057764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:57.057790 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:57.059157 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:57.059631 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:57.059636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:57.059637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:57.059742 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:57.060175 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:57.060178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:57.060179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:57.060425 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:57.060501 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:57.061180 (+ 679us) server_negotiation.cc:299] Negotiation successful
0809 09:29:57.061299 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 09:29:57.062692 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:57.050064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39097 (local address 127.9.102.3:57645)
0809 09:29:57.051185 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:57.051222 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:57.052179 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:57.052664 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:57.052669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:57.052925 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:29:57.053342 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:57.053364 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:57.054084 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:57.054087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:57.054957 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:57.054967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:57.055446 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:57.055450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:57.055617 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:57.056249 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:29:57.056272 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:29:57.057608 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:29:57.059270 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:57.059276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:57.059288 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:57.059522 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:57.059848 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:57.059852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:57.059854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:57.060040 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:57.060590 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:29:57.060596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:29:57.060894 (+ 298us) client_negotiation.cc:769] Sending connection context
0809 09:29:57.061648 (+ 754us) client_negotiation.cc:240] Negotiation successful
0809 09:29:57.061910 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":137,"threads_started":1}
I20260809 09:29:57.063750 31065 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39097
I20260809 09:29:57.063977 31065 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:57.064463 31065 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
I20260809 09:29:57.065436 30273 ts_manager.cc:194] Registered new tserver with Master: a30654a3df214f73ae1b58f560a910af (127.9.102.3:40951)
I20260809 09:29:57.066005 30273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:57645
I20260809 09:29:57.072898 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:29:57.073652 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 30411
I20260809 09:29:57.118425 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 30639
I20260809 09:29:57.129001 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 30852
I20260809 09:29:57.151196 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 30225
I20260809 09:29:57.161559 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39097
--webserver_interface=127.9.102.62
--webserver_port=33307
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39097
--ranger_config_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.102.20:60281/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.102.20:60281/kms with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:29:57.260955 31070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:29:57.261143 31070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:29:57.261175 31070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:29:57.264940 31070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:29:57.264988 31070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:29:57.265004 31070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:29:57.265018 31070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:29:57.265030 31070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:29:57.269402 31070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.102.20:60281/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39097
--ranger_config_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=33307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:29:57.270107 31070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:29:57.270870 31070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:29:57.278334 31077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:57.278364 31076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:57.278367 31084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:57.278669 31070 server_base.cc:1034] running on GCE node
I20260809 09:29:57.279157 31070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:29:57.280056 31070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:29:57.281246 31070 hybrid_clock.cc:648] HybridClock initialized: now 1786267797281234 us; error 30 us; skew 500 ppm
Aug 09 09:29:57 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267797, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:57.283638 31070 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:29:57.284425 31070 webserver.cc:460] Webserver started at http://127.9.102.62:33307/ using document root <none> and password file <none>
I20260809 09:29:57.285151 31070 fs_manager.cc:359] Metadata directory not provided
I20260809 09:29:57.285262 31070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 09:29:57 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267797, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:57.360713 31070 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 09:29:57.362814 31095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:57.363981 31070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:29:57.364153 31070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "afcc511c926d422a9cb976b267b80ccb"
format_stamp: "Formatted at 2026-08-09 09:29:55 on dist-test-slave-7vjv"
server_key: "bbb1222b0ff89e011cf3edc95dea1c84"
server_key_iv: "5f2c4f98dc92d1675e15f2bf5428e854"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:57.364806 31070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:29:57.385938 31070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:29:57.387990 31070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:29:57.388190 31070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:29:57.395062 31070 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:39097
I20260809 09:29:57.395071 31182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:39097 every 8 connection(s)
I20260809 09:29:57.396168 31070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:29:57.398608 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 31070
I20260809 09:29:57.399029 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.102.20:60281/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:38551
--local_ip_for_outbound_sockets=127.9.102.1
--tserver_master_addrs=127.9.102.62:39097
--webserver_port=35865
--webserver_interface=127.9.102.1
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.102.20:60281/kms with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:29:57.400430 31183 sys_catalog.cc:263] Verifying existing consensus state
I20260809 09:29:57.401978 31183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb: Bootstrap starting.
I20260809 09:29:57.418259 31183 log.cc:824] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb: Log is configured to *not* fsync() on all Append() calls
I20260809 09:29:57.425362 31183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 09:29:57.425766 31183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb: Bootstrap complete.
I20260809 09:29:57.429525 31183 raft_consensus.cc:348] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc511c926d422a9cb976b267b80ccb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39097 } }
I20260809 09:29:57.429921 31183 raft_consensus.cc:729] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: afcc511c926d422a9cb976b267b80ccb, State: Initialized, Role: FOLLOWER
I20260809 09:29:57.430400 31183 consensus_queue.cc:260] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc511c926d422a9cb976b267b80ccb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39097 } }
I20260809 09:29:57.430552 31183 raft_consensus.cc:388] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:29:57.430636 31183 raft_consensus.cc:482] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:29:57.430760 31183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 1 FOLLOWER]: Advancing to term 2
I20260809 09:29:57.432580 31183 raft_consensus.cc:504] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc511c926d422a9cb976b267b80ccb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39097 } }
I20260809 09:29:57.432909 31183 leader_election.cc:302] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afcc511c926d422a9cb976b267b80ccb; no voters:
I20260809 09:29:57.433197 31183 leader_election.cc:288] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 09:29:57.433310 31187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 2 FOLLOWER]: Leader election won for term 2
I20260809 09:29:57.433578 31187 raft_consensus.cc:686] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [term 2 LEADER]: Becoming Leader. State: Replica: afcc511c926d422a9cb976b267b80ccb, State: Running, Role: LEADER
I20260809 09:29:57.433977 31187 consensus_queue.cc:237] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc511c926d422a9cb976b267b80ccb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39097 } }
I20260809 09:29:57.434386 31183 sys_catalog.cc:564] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:29:57.434881 31187 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [sys.catalog]: SysCatalogTable state changed. Reason: New leader afcc511c926d422a9cb976b267b80ccb. Latest consensus state: current_term: 2 leader_uuid: "afcc511c926d422a9cb976b267b80ccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc511c926d422a9cb976b267b80ccb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39097 } } }
I20260809 09:29:57.435015 31187 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [sys.catalog]: This master's current role is: LEADER
I20260809 09:29:57.435384 31192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:29:57.435453 31188 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "afcc511c926d422a9cb976b267b80ccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc511c926d422a9cb976b267b80ccb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39097 } } }
I20260809 09:29:57.435654 31188 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb [sys.catalog]: This master's current role is: LEADER
I20260809 09:29:57.438412 31192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:29:57.438906 31183 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 09:29:57.439049 31192 catalog_manager.cc:1259] Loaded cluster ID: cbc9cab69bc94c84a4707a1b805c3a59
I20260809 09:29:57.439105 31192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:29:57.444478 31192 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:29:57.445973 31192 catalog_manager.cc:5879] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb: Loaded TSK: 0
I20260809 09:29:57.447311 31192 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 09:29:57.531967 31185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:29:57.532395 31185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:29:57.532517 31185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:29:57.537104 31185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:29:57.537191 31185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:29:57.537334 31185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:29:57.542456 31185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.102.20:60281/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:38551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=35865
--tserver_master_addrs=127.9.102.62:39097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:29:57.543394 31185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:29:57.544379 31185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:29:57.555397 31249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:57.555876 31250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:57.556216 31253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:57.557489 31185 server_base.cc:1034] running on GCE node
I20260809 09:29:57.557896 31185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:29:57.558616 31185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:29:57.559795 31185 hybrid_clock.cc:648] HybridClock initialized: now 1786267797559776 us; error 41 us; skew 500 ppm
Aug 09 09:29:57 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267797, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:57.562721 31185 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:29:57.563688 31185 webserver.cc:460] Webserver started at http://127.9.102.1:35865/ using document root <none> and password file <none>
I20260809 09:29:57.564306 31185 fs_manager.cc:359] Metadata directory not provided
I20260809 09:29:57.564428 31185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 09:29:57 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267797, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:57.590256 31185 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260809 09:29:57.594429 31266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:57.596040 31185 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 09:29:57.596208 31185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "85f4ac75f58b483d9aa14f073ff98183"
format_stamp: "Formatted at 2026-08-09 09:29:56 on dist-test-slave-7vjv"
server_key: "b446ac2cbd31a7f1f40f4e1a291e73ca"
server_key_iv: "222d71eb7b3a3a6298271b173d0b2608"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:57.597004 31185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:29:57.615007 31185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:29:57.617766 31185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:29:57.617955 31185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:29:57.618570 31185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:29:57.619429 31185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:29:57.619503 31185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:57.619614 31185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:29:57.619657 31185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:57.629793 31185 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:38551
I20260809 09:29:57.630909 31185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:29:57.632406 31454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:38551 every 8 connection(s)
I20260809 09:29:57.638602 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 31185
I20260809 09:29:57.639106 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.102.20:60281/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:46333
--local_ip_for_outbound_sockets=127.9.102.2
--tserver_master_addrs=127.9.102.62:39097
--webserver_port=45893
--webserver_interface=127.9.102.2
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.102.20:60281/kms with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 09:29:57 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267797, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:29:57.650604 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:57.633006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39097 (local address 127.9.102.1:42679)
0809 09:29:57.633444 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:57.633492 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:57.634473 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:57.636328 (+ 1855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:57.636337 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:57.636678 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:29:57.637320 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:57.637340 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:57.639409 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:57.639412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:57.640713 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:57.640724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:57.642454 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:57.642458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:57.642639 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:57.643193 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:29:57.643210 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:29:57.644908 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:29:57.647281 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:57.647286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:57.647297 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:57.647638 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:57.648056 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:57.648059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:57.648061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:57.648258 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:57.648999 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:29:57.649004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:29:57.649276 (+ 272us) client_negotiation.cc:769] Sending connection context
0809 09:29:57.649453 (+ 177us) client_negotiation.cc:240] Negotiation successful
0809 09:29:57.649768 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":93,"threads_started":1}
I20260809 09:29:57.650592 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:57.634034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:42679 (local address 127.9.102.62:39097)
0809 09:29:57.634992 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:57.635020 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:57.635083 (+ 63us) server_negotiation.cc:407] Connection header received
0809 09:29:57.635420 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:57.635437 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:57.635831 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:57.637571 (+ 1740us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:57.637866 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:57.639239 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:57.641621 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:57.642275 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:57.642452 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:57.645166 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:57.645172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:57.645187 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:57.645216 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:57.647138 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:57.647781 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:57.647788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:57.647794 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:57.647927 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:57.648410 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:57.648414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:57.648415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:57.648813 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:57.648939 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:57.649610 (+ 671us) server_negotiation.cc:299] Negotiation successful
0809 09:29:57.649828 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":98,"threads_started":1}
I20260809 09:29:57.653441 31455 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39097
I20260809 09:29:57.653679 31455 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:57.654175 31455 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
W20260809 09:29:57.656280 31455 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:39097 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 09:29:57.656390 31455 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:57.656548 31455 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
I20260809 09:29:57.657480 31455 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:57.657640 31455 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
W20260809 09:29:57.658440 31455 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 09:29:57.771492 31460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:29:57.771747 31460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:29:57.771836 31460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:29:57.776181 31460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:29:57.776253 31460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:29:57.776340 31460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:29:57.781549 31460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.102.20:60281/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:46333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=45893
--tserver_master_addrs=127.9.102.62:39097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:29:57.782524 31460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:29:57.783453 31460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:29:57.793058 31469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:57.793095 31471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:57.793380 31460 server_base.cc:1034] running on GCE node
W20260809 09:29:57.794682 31473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:57.795146 31460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:29:57.795850 31460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:29:57.797103 31460 hybrid_clock.cc:648] HybridClock initialized: now 1786267797797065 us; error 56 us; skew 500 ppm
Aug 09 09:29:57 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267797, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:57.800783 31460 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:29:57.801779 31460 webserver.cc:460] Webserver started at http://127.9.102.2:45893/ using document root <none> and password file <none>
I20260809 09:29:57.802439 31460 fs_manager.cc:359] Metadata directory not provided
I20260809 09:29:57.802542 31460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 09:29:57 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267797, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:57.819095 31460 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 09:29:57.821594 31488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:57.822798 31460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 09:29:57.822896 31460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "3a0a244e7a2e4d869e4f2deb6971cf30"
format_stamp: "Formatted at 2026-08-09 09:29:56 on dist-test-slave-7vjv"
server_key: "2dbac31589d9dc57e82c6e7f9935f8cf"
server_key_iv: "cfccb1a477034be9e5dc0db58d8e2138"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:57.823406 31460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:29:57.840425 31460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:29:57.847460 31460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:29:57.847677 31460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:29:57.848313 31460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:29:57.849305 31460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:29:57.849408 31460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:57.849498 31460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:29:57.849555 31460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:57.859742 31460 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:46333
I20260809 09:29:57.861027 31460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:29:57.861982 31672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:46333 every 8 connection(s)
I20260809 09:29:57.871218 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 31460
I20260809 09:29:57.871701 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.102.20:60281/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:40951
--local_ip_for_outbound_sockets=127.9.102.3
--tserver_master_addrs=127.9.102.62:39097
--webserver_port=44877
--webserver_interface=127.9.102.3
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:42963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.102.20:60281/kms with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 09:29:57 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267797, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:29:57.879325 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:57.864554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39097 (local address 127.9.102.2:54925)
0809 09:29:57.865599 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:57.865645 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:57.866635 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:57.867411 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:57.867419 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:57.867708 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:29:57.868421 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:57.868440 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:57.869595 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:57.869598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:57.870851 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:57.870871 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:57.871627 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:57.871630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:57.872010 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:57.872574 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:29:57.872592 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:29:57.874225 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:29:57.876448 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:57.876453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:57.876465 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:57.876745 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:57.877053 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:57.877056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:57.877058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:57.877208 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:57.877669 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:29:57.877674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:29:57.877979 (+ 305us) client_negotiation.cc:769] Sending connection context
0809 09:29:57.878150 (+ 171us) client_negotiation.cc:240] Negotiation successful
0809 09:29:57.878425 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":131,"threads_started":1}
I20260809 09:29:57.879873 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:57.864088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:54925 (local address 127.9.102.62:39097)
0809 09:29:57.864256 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:57.864260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:57.865736 (+ 1476us) server_negotiation.cc:407] Connection header received
0809 09:29:57.866893 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:57.866896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:57.866950 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:57.866997 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:57.868573 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:57.869398 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:57.870990 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:57.871326 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:57.871401 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:57.874395 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:57.874398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:57.874401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:57.874443 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:57.876304 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:57.876869 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:57.876872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:57.876874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:57.876955 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:57.877327 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:57.877330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:57.877332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:57.877532 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:57.877618 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:57.879571 (+ 1953us) server_negotiation.cc:299] Negotiation successful
0809 09:29:57.879702 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 09:29:57.881316 31674 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39097
I20260809 09:29:57.881511 31674 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:57.881973 31674 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
W20260809 09:29:57.883430 31674 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:39097 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 09:29:57.883538 31674 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:57.883687 31674 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
I20260809 09:29:57.884461 31674 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:57.884611 31674 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
W20260809 09:29:57.885387 31674 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 09:29:58.017266 31677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:29:58.017513 31677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:29:58.017606 31677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:29:58.021930 31677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:29:58.021993 31677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:29:58.022063 31677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:29:58.028165 31677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:42963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.102.20:60281/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:40951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=44877
--tserver_master_addrs=127.9.102.62:39097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:29:58.029531 31677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:29:58.030777 31677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:29:58.039650 31695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:58.039983 31689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:29:58.039702 31690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:29:58.040253 31677 server_base.cc:1034] running on GCE node
I20260809 09:29:58.040741 31677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:29:58.041481 31677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:29:58.048906 31677 hybrid_clock.cc:648] HybridClock initialized: now 1786267798048862 us; error 56 us; skew 500 ppm
Aug 09 09:29:58 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267798, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:58.052428 31677 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:29:58.053442 31677 webserver.cc:460] Webserver started at http://127.9.102.3:44877/ using document root <none> and password file <none>
I20260809 09:29:58.054085 31677 fs_manager.cc:359] Metadata directory not provided
I20260809 09:29:58.054186 31677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 09:29:58 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267798, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:58.080029 31677 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.000s
I20260809 09:29:58.082775 31711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:58.085599 31677 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 09:29:58.085747 31677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "a30654a3df214f73ae1b58f560a910af"
format_stamp: "Formatted at 2026-08-09 09:29:56 on dist-test-slave-7vjv"
server_key: "b0358c5ed8c2e2d44b44d0ce8caae673"
server_key_iv: "567523dde30036f90d8fb99bef17bd01"
server_key_version: "kuduclusterkey@0"
I20260809 09:29:58.086400 31677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:29:58.097873 31677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:29:58.100699 31677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:29:58.100909 31677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:29:58.101449 31677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:29:58.102320 31677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:29:58.102414 31677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:58.102492 31677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:29:58.102546 31677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:29:58.112563 31677 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:40951
I20260809 09:29:58.113796 31677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:29:58.115545 31895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:40951 every 8 connection(s)
I20260809 09:29:58.116673 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 31677
I20260809 09:29:58.130201 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:58.119354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.9.102.62:39097)
0809 09:29:58.120279 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:58.120283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:58.120304 (+ 21us) server_negotiation.cc:407] Connection header received
0809 09:29:58.120341 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:58.120343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:58.120379 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:58.120479 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:58.121605 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:58.122152 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:58.123679 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:58.123881 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:58.123977 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:58.127180 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:58.127183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:58.127185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:58.127204 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:58.128627 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:58.129023 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:58.129026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:58.129028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:58.129108 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:58.129360 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:58.129363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:58.129365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:58.129551 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:58.129740 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:58.129882 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 09:29:58.129993 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":96,"threads_started":1}
Aug 09 09:29:58 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267798, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:29:58.135387 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:58.119184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:53069 (local address 127.9.102.62:39097)
0809 09:29:58.119318 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:58.119322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:58.124178 (+ 4856us) server_negotiation.cc:407] Connection header received
0809 09:29:58.125296 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:58.125300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:58.125348 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:58.125411 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:58.126867 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:58.127537 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:58.129031 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:58.129314 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:58.129395 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:58.132206 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:58.132209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:58.132211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:58.132243 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:58.133488 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:58.133953 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:58.133956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:58.133958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:58.134039 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:58.134353 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:58.134356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:58.134358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:58.134529 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:58.134629 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:58.135107 (+ 478us) server_negotiation.cc:299] Negotiation successful
0809 09:29:58.135241 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53,"spinlock_wait_cycles":46848}
I20260809 09:29:58.136236 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:58.117640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39097 (local address 127.9.102.3:53069)
0809 09:29:58.124022 (+ 6382us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:58.124074 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:58.125065 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:58.125812 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:58.125820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:58.126137 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:29:58.126720 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:58.126737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:58.127689 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:58.127692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:58.128873 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:58.128884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:58.129597 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:58.129601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:58.129770 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:58.130290 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:29:58.130307 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:29:58.132025 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:29:58.133606 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:58.133611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:58.133623 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:58.133851 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:58.134125 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:29:58.134128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:29:58.134130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:29:58.134258 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:29:58.134633 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:29:58.134641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:29:58.134872 (+ 231us) client_negotiation.cc:769] Sending connection context
0809 09:29:58.135040 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 09:29:58.135315 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6186,"thread_start_us":134,"threads_started":1}
I20260809 09:29:58.137336 31900 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39097
I20260809 09:29:58.137575 31900 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:58.138059 31900 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
W20260809 09:29:58.139639 31900 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:39097 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 09:29:58.139770 31900 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:58.139935 31900 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
I20260809 09:29:58.140767 31900 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:58.140931 31900 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
W20260809 09:29:58.141709 31900 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 09:29:58 dist-test-slave-7vjv krb5kdc[29472](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786267798, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:29:58.658653 31455 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:58.658937 31455 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
I20260809 09:29:58.660274 31125 ts_manager.cc:194] Registered new tserver with Master: 85f4ac75f58b483d9aa14f073ff98183 (127.9.102.1:38551)
I20260809 09:29:58.662009 31125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:42679
Aug 09 09:29:58 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267798, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:58.885673 31674 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:58.885941 31674 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
I20260809 09:29:58.886646 31125 ts_manager.cc:194] Registered new tserver with Master: 3a0a244e7a2e4d869e4f2deb6971cf30 (127.9.102.2:46333)
I20260809 09:29:58.887368 31125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:54925
Aug 09 09:29:59 dist-test-slave-7vjv krb5kdc[29472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267798, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for HTTP/127.9.102.20@KRBTEST.COM
I20260809 09:29:59.141940 31900 heartbeater.cc:464] Registering TS with master...
I20260809 09:29:59.142241 31900 heartbeater.cc:511] Master 127.9.102.62:39097 requested a full tablet report, sending...
I20260809 09:29:59.142887 31125 ts_manager.cc:194] Registered new tserver with Master: a30654a3df214f73ae1b58f560a910af (127.9.102.3:40951)
I20260809 09:29:59.143664 31125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:53069
I20260809 09:29:59.152248 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:29:59.166518 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.153847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.9.102.62:39097)
0809 09:29:59.155453 (+ 1606us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:59.155458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:59.155471 (+ 13us) server_negotiation.cc:407] Connection header received
0809 09:29:59.155510 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:59.155514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:59.155565 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:59.155650 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:29:59.156648 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.157719 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.158880 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.159238 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.159338 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:59.161240 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:29:59.161244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:29:59.161246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:29:59.161271 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:29:59.164375 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:59.164861 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:59.164867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:59.164869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:59.164968 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:29:59.165285 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:29:59.165288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:29:59.165290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:29:59.165499 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:29:59.165627 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:59.166223 (+ 596us) server_negotiation.cc:299] Negotiation successful
0809 09:29:59.166353 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1536,"thread_start_us":82,"threads_started":1}
I20260809 09:29:59.171680 31125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:29:59.175699 31125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:29:59.205013 31959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.191178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38551 (local address 127.0.0.1:38118)
0809 09:29:59.195432 (+ 4254us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:59.195442 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:59.195540 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:59.198986 (+ 3446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:59.198988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:59.199021 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:29:59.199165 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:59.199175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.201182 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.201185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:59.202548 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:59.202558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.204382 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.204385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:59.204595 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:59.204637 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 09:29:59.204726 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 09:29:59.204821 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4202,"mutex_wait_us":149,"spinlock_wait_cycles":6144,"thread_start_us":71,"threads_started":1}
I20260809 09:29:59.205822 31964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.192762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.9.102.1:38551)
0809 09:29:59.193097 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:59.193103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:59.195540 (+ 2437us) server_negotiation.cc:407] Connection header received
0809 09:29:59.195650 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:59.195655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:59.195931 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:59.198801 (+ 2870us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:29:59.199311 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.201056 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.202688 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.203503 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.203609 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:59.204488 (+ 879us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:59.205586 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0809 09:29:59.205678 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":59,"threads_started":1}
I20260809 09:29:59.208690 31345 tablet_service.cc:1467] Processing CreateTablet for tablet 6804f57a79b14fe5b00182bec825fdaa (DEFAULT_TABLE table=test-table [id=151e91fc85834fbcbd02d867e3e8fd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:29:59.209777 31345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6804f57a79b14fe5b00182bec825fdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:29:59.214871 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.191008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:46333 (local address 127.0.0.1:58968)
0809 09:29:59.199129 (+ 8121us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:59.199141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:59.199243 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:59.201470 (+ 2227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:59.201473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:59.201490 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:29:59.201559 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:59.201565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.203519 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.203523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:59.204484 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:59.204493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.214468 (+ 9975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.214472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:59.214565 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:59.214606 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 09:29:59.214668 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 09:29:59.214729 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8072,"mutex_wait_us":1723,"thread_start_us":82,"threads_started":1}
I20260809 09:29:59.215907 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.191008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:40951 (local address 127.0.0.1:36646)
0809 09:29:59.195139 (+ 4131us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:59.195174 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:59.195269 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:59.198976 (+ 3707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:59.198986 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:59.199020 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:29:59.199916 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:59.199923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.201182 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.201185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:59.204859 (+ 3674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:59.204867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.215506 (+ 10639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.215509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:59.215611 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:59.215632 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 09:29:59.215702 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 09:29:59.215777 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3991,"mutex_wait_us":2128,"spinlock_wait_cycles":4608,"thread_start_us":99,"threads_started":1}
I20260809 09:29:59.216750 31961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.192272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.9.102.3:40951)
0809 09:29:59.196086 (+ 3814us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:59.196107 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:59.196215 (+ 108us) server_negotiation.cc:407] Connection header received
0809 09:29:59.196289 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:59.196293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:59.196470 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:59.196626 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:29:59.200045 (+ 3419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.201077 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.206043 (+ 4966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.215187 (+ 9144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.215299 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:59.215440 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:59.216511 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0809 09:29:59.216610 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3716,"thread_start_us":129,"threads_started":1}
I20260809 09:29:59.217309 31960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.191893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.9.102.2:46333)
0809 09:29:59.196177 (+ 4284us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:59.196184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:59.199327 (+ 3143us) server_negotiation.cc:407] Connection header received
0809 09:29:59.199403 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:59.199408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:59.199582 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:59.199721 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:29:59.201683 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.202678 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.204980 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.205753 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.205852 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:59.205989 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:59.217076 (+ 11087us) server_negotiation.cc:299] Negotiation successful
0809 09:29:59.217173 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3777,"thread_start_us":116,"threads_started":1}
I20260809 09:29:59.220425 31562 tablet_service.cc:1467] Processing CreateTablet for tablet 6804f57a79b14fe5b00182bec825fdaa (DEFAULT_TABLE table=test-table [id=151e91fc85834fbcbd02d867e3e8fd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:29:59.220412 31772 tablet_service.cc:1467] Processing CreateTablet for tablet 6804f57a79b14fe5b00182bec825fdaa (DEFAULT_TABLE table=test-table [id=151e91fc85834fbcbd02d867e3e8fd15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:29:59.221428 31772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6804f57a79b14fe5b00182bec825fdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:29:59.221436 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6804f57a79b14fe5b00182bec825fdaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:29:59.228475 31965 tablet_bootstrap.cc:492] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183: Bootstrap starting.
I20260809 09:29:59.230464 31966 tablet_bootstrap.cc:492] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30: Bootstrap starting.
I20260809 09:29:59.230464 31967 tablet_bootstrap.cc:492] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af: Bootstrap starting.
I20260809 09:29:59.230791 31965 tablet_bootstrap.cc:654] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183: Neither blocks nor log segments found. Creating new log.
I20260809 09:29:59.232543 31966 tablet_bootstrap.cc:654] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30: Neither blocks nor log segments found. Creating new log.
I20260809 09:29:59.232615 31967 tablet_bootstrap.cc:654] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af: Neither blocks nor log segments found. Creating new log.
I20260809 09:29:59.232944 31965 log.cc:824] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183: Log is configured to *not* fsync() on all Append() calls
I20260809 09:29:59.233407 31967 log.cc:824] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af: Log is configured to *not* fsync() on all Append() calls
I20260809 09:29:59.233407 31966 log.cc:824] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30: Log is configured to *not* fsync() on all Append() calls
I20260809 09:29:59.237077 31967 tablet_bootstrap.cc:492] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af: No bootstrap required, opened a new log
I20260809 09:29:59.237282 31967 ts_tablet_manager.cc:1397] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260809 09:29:59.241050 31967 raft_consensus.cc:348] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 46333 } } peers { permanent_uuid: "85f4ac75f58b483d9aa14f073ff98183" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38551 } } peers { permanent_uuid: "a30654a3df214f73ae1b58f560a910af" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 40951 } }
I20260809 09:29:59.241421 31965 tablet_bootstrap.cc:492] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183: No bootstrap required, opened a new log
I20260809 09:29:59.241410 31967 raft_consensus.cc:374] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:29:59.241495 31967 raft_consensus.cc:729] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a30654a3df214f73ae1b58f560a910af, State: Initialized, Role: FOLLOWER
I20260809 09:29:59.241691 31965 ts_tablet_manager.cc:1397] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260809 09:29:59.242056 31967 consensus_queue.cc:260] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 46333 } } peers { permanent_uuid: "85f4ac75f58b483d9aa14f073ff98183" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38551 } } peers { permanent_uuid: "a30654a3df214f73ae1b58f560a910af" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 40951 } }
I20260809 09:29:59.243116 31967 ts_tablet_manager.cc:1428] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 09:29:59.243494 31900 heartbeater.cc:503] Master 127.9.102.62:39097 was elected leader, sending a full tablet report...
I20260809 09:29:59.243690 31966 tablet_bootstrap.cc:492] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30: No bootstrap required, opened a new log
I20260809 09:29:59.243851 31966 ts_tablet_manager.cc:1397] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260809 09:29:59.245775 31965 raft_consensus.cc:348] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 46333 } } peers { permanent_uuid: "85f4ac75f58b483d9aa14f073ff98183" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38551 } } peers { permanent_uuid: "a30654a3df214f73ae1b58f560a910af" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 40951 } }
I20260809 09:29:59.246001 31965 raft_consensus.cc:374] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:29:59.246048 31965 raft_consensus.cc:729] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f4ac75f58b483d9aa14f073ff98183, State: Initialized, Role: FOLLOWER
I20260809 09:29:59.246467 31965 consensus_queue.cc:260] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 46333 } } peers { permanent_uuid: "85f4ac75f58b483d9aa14f073ff98183" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38551 } } peers { permanent_uuid: "a30654a3df214f73ae1b58f560a910af" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 40951 } }
I20260809 09:29:59.247349 31965 ts_tablet_manager.cc:1428] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 09:29:59.247931 31966 raft_consensus.cc:348] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 46333 } } peers { permanent_uuid: "85f4ac75f58b483d9aa14f073ff98183" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38551 } } peers { permanent_uuid: "a30654a3df214f73ae1b58f560a910af" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 40951 } }
I20260809 09:29:59.248248 31966 raft_consensus.cc:374] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:29:59.248343 31966 raft_consensus.cc:729] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a0a244e7a2e4d869e4f2deb6971cf30, State: Initialized, Role: FOLLOWER
I20260809 09:29:59.248790 31966 consensus_queue.cc:260] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 46333 } } peers { permanent_uuid: "85f4ac75f58b483d9aa14f073ff98183" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38551 } } peers { permanent_uuid: "a30654a3df214f73ae1b58f560a910af" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 40951 } }
I20260809 09:29:59.249725 31966 ts_tablet_manager.cc:1428] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260809 09:29:59.250102 31674 heartbeater.cc:503] Master 127.9.102.62:39097 was elected leader, sending a full tablet report...
I20260809 09:29:59.252386 31455 heartbeater.cc:503] Master 127.9.102.62:39097 was elected leader, sending a full tablet report...
I20260809 09:29:59.326198 31984 raft_consensus.cc:482] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:29:59.326465 31984 raft_consensus.cc:504] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 46333 } } peers { permanent_uuid: "85f4ac75f58b483d9aa14f073ff98183" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38551 } } peers { permanent_uuid: "a30654a3df214f73ae1b58f560a910af" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 40951 } }
I20260809 09:29:59.327967 31984 leader_election.cc:288] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f4ac75f58b483d9aa14f073ff98183 (127.9.102.1:38551), a30654a3df214f73ae1b58f560a910af (127.9.102.3:40951)
I20260809 09:29:59.339627 31961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.328645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:50095 (local address 127.9.102.3:40951)
0809 09:29:59.328762 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:59.328766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:59.332346 (+ 3580us) server_negotiation.cc:407] Connection header received
0809 09:29:59.332571 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:59.332574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:59.332625 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:59.332670 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:29:59.334176 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.335014 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.337673 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.338215 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.338294 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:59.338374 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:59.339361 (+ 987us) server_negotiation.cc:299] Negotiation successful
0809 09:29:59.339461 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:29:59.340297 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.328257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:40951 (local address 127.9.102.2:50095)
0809 09:29:59.332246 (+ 3989us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:59.332261 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:59.332434 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:59.332804 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:59.332807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:59.332822 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:29:59.333114 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:59.333124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.335174 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.335178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:59.337525 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:59.337537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.338471 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.338474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:59.338576 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:59.338599 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 09:29:59.340044 (+ 1445us) client_negotiation.cc:240] Negotiation successful
0809 09:29:59.340147 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3922,"thread_start_us":123,"threads_started":1}
I20260809 09:29:59.341146 31818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6804f57a79b14fe5b00182bec825fdaa" candidate_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30654a3df214f73ae1b58f560a910af" is_pre_election: true
I20260809 09:29:59.341475 31818 raft_consensus.cc:2454] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a0a244e7a2e4d869e4f2deb6971cf30 in term 0.
I20260809 09:29:59.342072 31489 leader_election.cc:302] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a0a244e7a2e4d869e4f2deb6971cf30, a30654a3df214f73ae1b58f560a910af; no voters:
I20260809 09:29:59.342379 31984 raft_consensus.cc:2781] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:29:59.342458 31984 raft_consensus.cc:482] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:29:59.342499 31984 raft_consensus.cc:3037] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:29:59.344177 31984 raft_consensus.cc:504] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 46333 } } peers { permanent_uuid: "85f4ac75f58b483d9aa14f073ff98183" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38551 } } peers { permanent_uuid: "a30654a3df214f73ae1b58f560a910af" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 40951 } }
I20260809 09:29:59.344606 31984 leader_election.cc:288] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [CANDIDATE]: Term 1 election: Requested vote from peers 85f4ac75f58b483d9aa14f073ff98183 (127.9.102.1:38551), a30654a3df214f73ae1b58f560a910af (127.9.102.3:40951)
I20260809 09:29:59.345237 31818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6804f57a79b14fe5b00182bec825fdaa" candidate_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a30654a3df214f73ae1b58f560a910af"
I20260809 09:29:59.345402 31818 raft_consensus.cc:3037] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:29:59.346700 31818 raft_consensus.cc:2454] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a0a244e7a2e4d869e4f2deb6971cf30 in term 1.
I20260809 09:29:59.347069 31489 leader_election.cc:302] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a0a244e7a2e4d869e4f2deb6971cf30, a30654a3df214f73ae1b58f560a910af; no voters:
I20260809 09:29:59.347280 31984 raft_consensus.cc:2781] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:29:59.347497 31984 raft_consensus.cc:686] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 1 LEADER]: Becoming Leader. State: Replica: 3a0a244e7a2e4d869e4f2deb6971cf30, State: Running, Role: LEADER
I20260809 09:29:59.347827 31984 consensus_queue.cc:237] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 46333 } } peers { permanent_uuid: "85f4ac75f58b483d9aa14f073ff98183" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38551 } } peers { permanent_uuid: "a30654a3df214f73ae1b58f560a910af" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 40951 } }
I20260809 09:29:59.348490 31964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.328937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:57785 (local address 127.9.102.1:38551)
0809 09:29:59.329057 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:59.329061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:59.344442 (+ 15381us) server_negotiation.cc:407] Connection header received
0809 09:29:59.344671 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:59.344675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:59.344741 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:59.344801 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:29:59.345297 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.346009 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.347281 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.347782 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.347876 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:59.347940 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:59.348287 (+ 347us) server_negotiation.cc:299] Negotiation successful
0809 09:29:59.348369 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:29:59.348945 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.328391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38551 (local address 127.9.102.2:57785)
0809 09:29:59.344327 (+ 15936us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:59.344345 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 09:29:59.344521 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:29:59.344978 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:29:59.344983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:29:59.345005 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:29:59.345133 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:59.345142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.346163 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.346167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:59.347143 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:29:59.347152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.348033 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.348037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:29:59.348145 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:59.348168 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 09:29:59.348739 (+ 571us) client_negotiation.cc:240] Negotiation successful
0809 09:29:59.348804 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15867,"thread_start_us":106,"threads_started":1}
I20260809 09:29:59.349772 31375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6804f57a79b14fe5b00182bec825fdaa" candidate_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f4ac75f58b483d9aa14f073ff98183" is_pre_election: true
I20260809 09:29:59.350054 31375 raft_consensus.cc:2454] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a0a244e7a2e4d869e4f2deb6971cf30 in term 0.
I20260809 09:29:59.350400 31373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6804f57a79b14fe5b00182bec825fdaa" candidate_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f4ac75f58b483d9aa14f073ff98183"
I20260809 09:29:59.350507 31373 raft_consensus.cc:3037] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:29:59.351837 31373 raft_consensus.cc:2454] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a0a244e7a2e4d869e4f2deb6971cf30 in term 1.
I20260809 09:29:59.355155 31125 catalog_manager.cc:5495] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a0a244e7a2e4d869e4f2deb6971cf30 (127.9.102.2). New cstate: current_term: 1 leader_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a0a244e7a2e4d869e4f2deb6971cf30" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 46333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f4ac75f58b483d9aa14f073ff98183" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a30654a3df214f73ae1b58f560a910af" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 40951 } health_report { overall_health: UNKNOWN } } }
W20260809 09:29:59.363530 31675 tablet.cc:2369] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:29:59.370664 31902 tablet.cc:2369] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:29:59.383564 31456 tablet.cc:2369] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:29:59.394815 31960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.389647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.9.102.2:46333)
0809 09:29:59.389789 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0809 09:29:59.389794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:29:59.389813 (+ 19us) server_negotiation.cc:407] Connection header received
0809 09:29:59.389907 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:29:59.389910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:29:59.389993 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:29:59.390116 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:29:59.390621 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.391427 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.392878 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:29:59.393233 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:29:59.393370 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:29:59.393816 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:29:59.394361 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:29:59.394491 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 09:29:59.394584 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 09:29:59.394649 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 09:29:59.404207 31818 raft_consensus.cc:1261] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af [term 1 FOLLOWER]: Refusing update from remote peer 3a0a244e7a2e4d869e4f2deb6971cf30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:29:59.404341 31373 raft_consensus.cc:1261] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183 [term 1 FOLLOWER]: Refusing update from remote peer 3a0a244e7a2e4d869e4f2deb6971cf30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:29:59.404985 31987 consensus_queue.cc:1035] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f4ac75f58b483d9aa14f073ff98183" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:29:59.405143 31984 consensus_queue.cc:1035] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a30654a3df214f73ae1b58f560a910af" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:29:59.413383 32003 mvcc.cc:205] Tried to move back new op lower bound from 7316552906350645248 to 7316552906132660224. Current Snapshot: MvccSnapshot[applied={T|T < 7316552906350645248}]
I20260809 09:29:59.414443 32005 mvcc.cc:205] Tried to move back new op lower bound from 7316552906350645248 to 7316552906132660224. Current Snapshot: MvccSnapshot[applied={T|T < 7316552906350645248}]
I20260809 09:29:59.417485 32004 mvcc.cc:205] Tried to move back new op lower bound from 7316552906350645248 to 7316552906132660224. Current Snapshot: MvccSnapshot[applied={T|T < 7316552906350645248}]
I20260809 09:29:59.424418 31124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:29:59.424599 31124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:29:59.427569 31124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afcc511c926d422a9cb976b267b80ccb: Sending DeleteTablet for 3 replicas of tablet 6804f57a79b14fe5b00182bec825fdaa
I20260809 09:29:59.428299 31562 tablet_service.cc:1514] Processing DeleteTablet for tablet 6804f57a79b14fe5b00182bec825fdaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:29:59 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:58968
I20260809 09:29:59.428476 31345 tablet_service.cc:1514] Processing DeleteTablet for tablet 6804f57a79b14fe5b00182bec825fdaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:29:59 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:38118
I20260809 09:29:59.429196 31772 tablet_service.cc:1514] Processing DeleteTablet for tablet 6804f57a79b14fe5b00182bec825fdaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:29:59 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:36646
I20260809 09:29:59.429237 32017 tablet_replica.cc:330] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183: stopping tablet replica
I20260809 09:29:59.429531 32017 raft_consensus.cc:2229] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:29:59.429874 32017 raft_consensus.cc:2258] T 6804f57a79b14fe5b00182bec825fdaa P 85f4ac75f58b483d9aa14f073ff98183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:29:59.430325 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 31185
I20260809 09:29:59.431350 32015 tablet_replica.cc:330] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30: stopping tablet replica
I20260809 09:29:59.431655 32015 raft_consensus.cc:2229] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:29:59.432019 32015 raft_consensus.cc:2258] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:29:59.435176 32015 ts_tablet_manager.cc:1905] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:29:59.438304 32015 ts_tablet_manager.cc:1918] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:29:59.438591 32018 tablet_replica.cc:330] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af: stopping tablet replica
I20260809 09:29:59.438920 32018 raft_consensus.cc:2229] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:29:59.439280 32018 raft_consensus.cc:2258] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:29:59.439635 32015 log.cc:1196] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/6804f57a79b14fe5b00182bec825fdaa
I20260809 09:29:59.440064 32015 ts_tablet_manager.cc:1939] T 6804f57a79b14fe5b00182bec825fdaa P 3a0a244e7a2e4d869e4f2deb6971cf30: Deleting consensus metadata
I20260809 09:29:59.441279 31098 catalog_manager.cc:4841] TS 3a0a244e7a2e4d869e4f2deb6971cf30 (127.9.102.2:46333): tablet 6804f57a79b14fe5b00182bec825fdaa (table test-table [id=151e91fc85834fbcbd02d867e3e8fd15]) successfully deleted
I20260809 09:29:59.442878 32018 ts_tablet_manager.cc:1905] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:29:59.445779 32018 ts_tablet_manager.cc:1918] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260809 09:29:59.446098 31097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:38551 (error 108)
I20260809 09:29:59.446395 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 31460
I20260809 09:29:59.455981 32018 log.cc:1196] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/6804f57a79b14fe5b00182bec825fdaa
I20260809 09:29:59.456578 32018 ts_tablet_manager.cc:1939] T 6804f57a79b14fe5b00182bec825fdaa P a30654a3df214f73ae1b58f560a910af: Deleting consensus metadata
I20260809 09:29:59.458014 31096 catalog_manager.cc:4841] TS a30654a3df214f73ae1b58f560a910af (127.9.102.3:40951): tablet 6804f57a79b14fe5b00182bec825fdaa (table test-table [id=151e91fc85834fbcbd02d867e3e8fd15]) successfully deleted
I20260809 09:29:59.459175 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:29:59.458793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38551 (local address 127.0.0.1:37914)
0809 09:29:59.458979 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 09:29:59.459073 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:38551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260809 09:29:59.459450 31097 catalog_manager.cc:4584] TS 85f4ac75f58b483d9aa14f073ff98183 (127.9.102.1:38551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6804f57a79b14fe5b00182bec825fdaa: Network error: Client connection negotiation failed: client connection to 127.9.102.1:38551: connect: Connection refused (error 111)
I20260809 09:29:59.470887 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 31677
I20260809 09:29:59.482191 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 31070
I20260809 09:29:59.499326 9624 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 09:29:59.845009 9624 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 09:29:59.845150 9624 mini_ranger.cc:66] Stopping Ranger...
I20260809 09:30:00.202903 9624 mini_ranger.cc:68] Stopped Ranger
2026-08-09 09:30:00.203 UTC [29523] LOG: received smart shutdown request
2026-08-09 09:30:00.207 UTC [29523] LOG: background worker "logical replication launcher" (PID 29537) exited with exit code 1
2026-08-09 09:30:00.207 UTC [29530] LOG: shutting down
2026-08-09 09:30:00.346 UTC [29523] LOG: database system is shut down
2026-08-09T09:30:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45844 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:30:00 dist-test-slave-7vjv krb5kdc[32034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:30:00 dist-test-slave-7vjv krb5kdc[32034](info): set up 2 sockets
Aug 09 09:30:00 dist-test-slave-7vjv krb5kdc[32034](info): commencing operation
krb5kdc: starting...
W20260809 09:30:01.668428 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:30:01 dist-test-slave-7vjv krb5kdc[32034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:30:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:30:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:30:01.778630 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39769
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39769,127.9.102.61:38781,127.9.102.60:41293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:01.879269 32058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:01.879460 32058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:01.879508 32058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:01.883188 32058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:30:01.883229 32058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:01.883244 32058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:01.883257 32058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:30:01.883270 32058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:30:01.887745 32058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39769,127.9.102.61:38781,127.9.102.60:41293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:01.888561 32058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:01.889362 32058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:01.896852 32066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:01.896883 32068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:01.896981 32058 server_base.cc:1034] running on GCE node
W20260809 09:30:01.897156 32072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:01.897646 32058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:01.898579 32058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:01.899763 32058 hybrid_clock.cc:648] HybridClock initialized: now 1786267801899743 us; error 31 us; skew 500 ppm
Aug 09 09:30:01 dist-test-slave-7vjv krb5kdc[32034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267801, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:01.902418 32058 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:30:01.903227 32058 webserver.cc:460] Webserver started at http://127.9.102.62:43081/ using document root <none> and password file <none>
I20260809 09:30:01.903766 32058 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:01.903858 32058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:01.904070 32058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:30:01.905907 32058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "da3532e0130242dbad17f5ef81f0ef15"
format_stamp: "Formatted at 2026-08-09 09:30:01 on dist-test-slave-7vjv"
server_key: "5283a8517fd4306b58d145245579d521"
server_key_iv: "78f9567c024a9722ef9c7c089244c4fa"
server_key_version: "encryptionkey@0"
I20260809 09:30:01.906447 32058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "da3532e0130242dbad17f5ef81f0ef15"
format_stamp: "Formatted at 2026-08-09 09:30:01 on dist-test-slave-7vjv"
server_key: "5283a8517fd4306b58d145245579d521"
server_key_iv: "78f9567c024a9722ef9c7c089244c4fa"
server_key_version: "encryptionkey@0"
I20260809 09:30:01.909693 32058 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:30:01.911829 32085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:01.912808 32058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:01.912935 32058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "da3532e0130242dbad17f5ef81f0ef15"
format_stamp: "Formatted at 2026-08-09 09:30:01 on dist-test-slave-7vjv"
server_key: "5283a8517fd4306b58d145245579d521"
server_key_iv: "78f9567c024a9722ef9c7c089244c4fa"
server_key_version: "encryptionkey@0"
I20260809 09:30:01.913028 32058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:01.927258 32058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:01.929947 32058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:01.930142 32058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:01.936520 32058 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:39769
I20260809 09:30:01.936542 32171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:39769 every 8 connection(s)
I20260809 09:30:01.937330 32058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:30:01.939736 32172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:01.943233 32172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:30:01.944027 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 32058
I20260809 09:30:01.944164 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:30:01.944428 9624 external_mini_cluster.cc:1428] Setting key 78a9827b55fe1a4172fb6f0e7f53ff0b
Aug 09 09:30:01 dist-test-slave-7vjv krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267801, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.102.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.61@KRBTEST.COM" created.
I20260809 09:30:01.957862 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:01.945511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.9.102.62:39769)
0809 09:30:01.946014 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:01.946029 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:01.946183 (+ 154us) server_negotiation.cc:407] Connection header received
0809 09:30:01.947371 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:01.947378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:01.947625 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:01.947830 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:01.948643 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:01.949938 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:01.950881 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:01.951534 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:01.951752 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:01.953698 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:01.953704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:01.953715 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:01.953736 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:01.955217 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:01.955619 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:01.955624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:01.955625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:01.955717 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:01.955990 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:01.955992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:01.955993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:01.956464 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:01.956623 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:01.956991 (+ 368us) server_negotiation.cc:299] Negotiation successful
0809 09:30:01.957218 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":398,"spinlock_wait_cycles":45440,"thread_start_us":144,"threads_started":1}
I20260809 09:30:01.957866 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:01.945511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39769 (local address 127.0.0.1:36772)
0809 09:30:01.946029 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:01.946077 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:01.946924 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:01.947844 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:01.947850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:01.948280 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:01.948495 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:01.948507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:01.950112 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:01.950116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:01.950758 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:01.950766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:01.951695 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:01.951698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:01.951793 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:01.952317 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:01.952331 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:01.953561 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:01.955317 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:01.955325 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:01.955331 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:01.955532 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:01.955805 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:01.955812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:01.955814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:01.955913 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:01.956577 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:01.956581 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:01.956763 (+ 182us) client_negotiation.cc:769] Sending connection context
0809 09:30:01.956930 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 09:30:01.957332 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":83,"thread_start_us":143,"threads_started":1}
I20260809 09:30:01.960055 32172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:30:01.960807 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:01.960545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:38781 (local address 127.0.0.1:47236)
0809 09:30:01.960694 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:01.960763 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:38781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260809 09:30:01.960990 32088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.102.61:38781: connect: Connection refused (error 111)
I20260809 09:30:01.961705 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:01.961512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:38781 (local address 127.0.0.1:47246)
0809 09:30:01.961608 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:01.961665 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:38781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 09:30:01.961848 32172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.61:38781: Network error: Client connection negotiation failed: client connection to 127.9.102.61:38781: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.61.keytab.
Entry for principal kudu/127.9.102.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.61.keytab.
I20260809 09:30:01.970536 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.61:38781
--webserver_interface=127.9.102.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.61.keytab
--principal=kudu/127.9.102.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39769,127.9.102.61:38781,127.9.102.60:41293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:30:02.011163 32172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } attempt: 1
I20260809 09:30:02.012082 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.011733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:38781 (local address 127.0.0.1:47254)
0809 09:30:02.011902 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.011975 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:38781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 09:30:02.012890 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.012634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:38781 (local address 127.0.0.1:47262)
0809 09:30:02.012773 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.012823 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:38781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 09:30:02.013042 32172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.61:38781: Network error: Client connection negotiation failed: client connection to 127.9.102.61:38781: connect: Connection refused (error 111)
W20260809 09:30:02.072539 32184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:02.072754 32184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:02.072845 32184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:02.076426 32184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:30:02.076503 32184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:02.076546 32184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:02.076601 32184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:30:02.076637 32184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:30:02.081218 32184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39769,127.9.102.61:38781,127.9.102.60:41293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.61.keytab
--principal=kudu/127.9.102.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.61:38781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.102.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:02.081233 32172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } attempt: 2
I20260809 09:30:02.081992 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.081665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:38781 (local address 127.0.0.1:47278)
0809 09:30:02.081825 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.081910 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:38781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 09:30:02.082226 32184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:02.082734 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.082504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:38781 (local address 127.0.0.1:47290)
0809 09:30:02.082616 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.082665 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:38781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 09:30:02.082896 32172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.61:38781: Network error: Client connection negotiation failed: client connection to 127.9.102.61:38781: connect: Connection refused (error 111)
I20260809 09:30:02.083339 32184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:02.090631 32194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:02.090654 32199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:02.090664 32193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:02.090763 32184 server_base.cc:1034] running on GCE node
I20260809 09:30:02.091346 32184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:02.092298 32184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:02.093482 32184 hybrid_clock.cc:648] HybridClock initialized: now 1786267802093466 us; error 24 us; skew 500 ppm
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267802, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:02.102366 32184 init.cc:377] Logged in from keytab as kudu/127.9.102.61@KRBTEST.COM (short username kudu)
I20260809 09:30:02.103091 32184 webserver.cc:460] Webserver started at http://127.9.102.61:38381/ using document root <none> and password file <none>
I20260809 09:30:02.103542 32184 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:02.103592 32184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:02.103726 32184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:30:02.105540 32184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/data/instance:
uuid: "30cbc3b5ef984e19a4087f4515feb8e6"
format_stamp: "Formatted at 2026-08-09 09:30:02 on dist-test-slave-7vjv"
server_key: "dc51321b2c4ff51393667d865c279dc7"
server_key_iv: "05d6fe4662ec23396290225e2c90cb5a"
server_key_version: "encryptionkey@0"
I20260809 09:30:02.106086 32184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/wal/instance:
uuid: "30cbc3b5ef984e19a4087f4515feb8e6"
format_stamp: "Formatted at 2026-08-09 09:30:02 on dist-test-slave-7vjv"
server_key: "dc51321b2c4ff51393667d865c279dc7"
server_key_iv: "05d6fe4662ec23396290225e2c90cb5a"
server_key_version: "encryptionkey@0"
I20260809 09:30:02.109633 32184 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:30:02.111634 32207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:02.112733 32184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:02.112859 32184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/wal
uuid: "30cbc3b5ef984e19a4087f4515feb8e6"
format_stamp: "Formatted at 2026-08-09 09:30:02 on dist-test-slave-7vjv"
server_key: "dc51321b2c4ff51393667d865c279dc7"
server_key_iv: "05d6fe4662ec23396290225e2c90cb5a"
server_key_version: "encryptionkey@0"
I20260809 09:30:02.112967 32184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:02.127262 32184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:02.130147 32184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:02.130272 32184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:02.136754 32184 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.61:38781
I20260809 09:30:02.136770 32293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.61:38781 every 8 connection(s)
I20260809 09:30:02.137607 32184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
I20260809 09:30:02.140225 32294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:02.143559 32294 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:30:02.145857 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 32184
I20260809 09:30:02.145983 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-1/wal/instance
I20260809 09:30:02.146250 9624 external_mini_cluster.cc:1428] Setting key f67b18310665df39b94c57ac760db7ed
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267802, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.61@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.102.60@KRBTEST.COM; defaulting to no policy
I20260809 09:30:02.156883 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.145231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.9.102.62:39769)
0809 09:30:02.145364 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.145368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.146194 (+ 826us) server_negotiation.cc:407] Connection header received
0809 09:30:02.147096 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.147099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.147153 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.147251 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.148498 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.148995 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.150108 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.150341 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.150416 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.153010 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.153013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.153015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.153037 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.154929 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.155398 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.155402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.155404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.155489 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.155800 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.155805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.155808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.155969 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.156073 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.156585 (+ 512us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.156697 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 09:30:02.157518 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.145535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39769 (local address 127.0.0.1:36786)
0809 09:30:02.146046 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.146102 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:02.146898 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:02.147410 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:02.147415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:02.147705 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:02.148331 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.148354 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.149130 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.149135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.149989 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.149999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.150485 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.150492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.150713 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.151240 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:02.151263 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:02.152852 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:02.155047 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.155052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.155063 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.155300 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.155579 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.155582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.155584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.155710 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.156071 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:02.156078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:02.156319 (+ 241us) client_negotiation.cc:769] Sending connection context
0809 09:30:02.156488 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 09:30:02.156772 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.60@KRBTEST.COM" created.
I20260809 09:30:02.158828 32294 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267802, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.61@KRBTEST.COM for kudu/127.9.102.61@KRBTEST.COM
I20260809 09:30:02.166801 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.159275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:38781 (local address 127.0.0.1:47300)
0809 09:30:02.159375 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.159389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:02.159456 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:02.159978 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:02.159981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:02.160169 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:02.160239 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.160244 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.161089 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.161094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.161544 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.161551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.162073 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.162076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.162187 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.162571 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:02.162583 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:02.163587 (+ 1004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:02.165303 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.165306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.165308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.165490 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.165769 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.165773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.165775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.165851 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.166336 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:02.166339 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:02.166419 (+ 80us) client_negotiation.cc:769] Sending connection context
0809 09:30:02.166535 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 09:30:02.166647 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":12}
I20260809 09:30:02.166859 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.159396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.9.102.61:38781)
0809 09:30:02.159695 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.159702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.159718 (+ 16us) server_negotiation.cc:407] Connection header received
0809 09:30:02.159748 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.159751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.159869 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.159996 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.160346 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.160975 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.161655 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.161969 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.162075 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.163700 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.163715 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.163724 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.163740 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.165187 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.165603 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.165615 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.165616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.165681 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.165938 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.165941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.165942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.166229 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.166354 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.166600 (+ 246us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.166706 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":226,"thread_start_us":139,"threads_started":1}
I20260809 09:30:02.168345 32294 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:30:02.168973 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.168754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58170)
0809 09:30:02.168855 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.168923 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 09:30:02.169145 32210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
I20260809 09:30:02.169914 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.169729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58178)
0809 09:30:02.169823 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.169865 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260809 09:30:02.170080 32294 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:41293: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.60.keytab.
Entry for principal kudu/127.9.102.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.60.keytab.
I20260809 09:30:02.172566 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.60:41293
--webserver_interface=127.9.102.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.60.keytab
--principal=kudu/127.9.102.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39769,127.9.102.61:38781,127.9.102.60:41293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:30:02.174077 32172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } attempt: 3
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267801, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.61@KRBTEST.COM
I20260809 09:30:02.182725 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.174452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:38781 (local address 127.0.0.1:47310)
0809 09:30:02.174570 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.174585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:02.174675 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:02.174902 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:02.174904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:02.175060 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:02.175147 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.175152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.176441 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.176445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.176924 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.176932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.177356 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.177360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.177460 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.177856 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:02.177866 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:02.179282 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:02.181049 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.181055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.181058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.181346 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.181668 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.181671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.181672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.181775 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.182229 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:02.182232 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:02.182360 (+ 128us) client_negotiation.cc:769] Sending connection context
0809 09:30:02.182458 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 09:30:02.182588 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 09:30:02.182979 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.174503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.9.102.61:38781)
0809 09:30:02.174587 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.174591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.174629 (+ 38us) server_negotiation.cc:407] Connection header received
0809 09:30:02.174763 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.174767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.174809 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.174885 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.175253 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.176341 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.177042 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.177249 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.177345 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.179408 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.179411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.179412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.179430 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.180932 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.181471 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.181475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.181477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.181567 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.181892 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.181894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.181896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.182121 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.182208 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.182650 (+ 442us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.182744 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 09:30:02.183468 32172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:30:02.184049 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.183860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58182)
0809 09:30:02.183949 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.183992 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260809 09:30:02.185631 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.185308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58186)
0809 09:30:02.185474 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.185557 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260809 09:30:02.185745 32172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:41293: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
I20260809 09:30:02.216869 32172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } attempt: 1
I20260809 09:30:02.217736 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.217363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58200)
0809 09:30:02.217571 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.217631 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260809 09:30:02.218753 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.218311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58216)
0809 09:30:02.218629 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.218681 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
W20260809 09:30:02.218896 32172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:41293: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
I20260809 09:30:02.219331 32294 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } attempt: 1
I20260809 09:30:02.220010 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.219724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58226)
0809 09:30:02.219887 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.219936 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 09:30:02.220734 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.220480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58230)
0809 09:30:02.220639 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.220682 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260809 09:30:02.220911 32294 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:41293: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
I20260809 09:30:02.289070 32294 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } attempt: 2
I20260809 09:30:02.289866 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.289596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58244)
0809 09:30:02.289739 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.289795 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 09:30:02.290556 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.290353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58250)
0809 09:30:02.290451 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.290499 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 09:30:02.290683 32294 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:41293: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
I20260809 09:30:02.294064 32172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } attempt: 2
I20260809 09:30:02.294741 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.294441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58254)
0809 09:30:02.294601 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.294666 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 09:30:02.295485 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.295212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58266)
0809 09:30:02.295330 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.295399 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 09:30:02.295637 32172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:41293: Network error: Client connection negotiation failed: client connection to 127.9.102.60:41293: connect: Connection refused (error 111)
W20260809 09:30:02.300019 32310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:02.300289 32310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:02.300380 32310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:02.303931 32310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:30:02.303977 32310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:02.304016 32310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:02.304050 32310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:30:02.304116 32310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:30:02.308943 32310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39769,127.9.102.61:38781,127.9.102.60:41293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.60.keytab
--principal=kudu/127.9.102.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.60:41293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.102.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:02.309780 32310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:02.310674 32310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:02.318658 32330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:02.318658 32332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:02.318715 32328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:02.318850 32310 server_base.cc:1034] running on GCE node
I20260809 09:30:02.319465 32310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:02.320537 32310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:02.321717 32310 hybrid_clock.cc:648] HybridClock initialized: now 1786267802321709 us; error 25 us; skew 500 ppm
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267802, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:02.324621 32310 init.cc:377] Logged in from keytab as kudu/127.9.102.60@KRBTEST.COM (short username kudu)
I20260809 09:30:02.325348 32310 webserver.cc:460] Webserver started at http://127.9.102.60:42891/ using document root <none> and password file <none>
I20260809 09:30:02.325831 32310 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:02.325933 32310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:02.326193 32310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:30:02.327961 32310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/data/instance:
uuid: "63b152680487497b80946acedb5ae7e4"
format_stamp: "Formatted at 2026-08-09 09:30:02 on dist-test-slave-7vjv"
server_key: "9d79e1cab00a908248f78c6b96af18af"
server_key_iv: "9a73bbb5d4ccaddec77098749defbb3d"
server_key_version: "encryptionkey@0"
I20260809 09:30:02.328562 32310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/wal/instance:
uuid: "63b152680487497b80946acedb5ae7e4"
format_stamp: "Formatted at 2026-08-09 09:30:02 on dist-test-slave-7vjv"
server_key: "9d79e1cab00a908248f78c6b96af18af"
server_key_iv: "9a73bbb5d4ccaddec77098749defbb3d"
server_key_version: "encryptionkey@0"
I20260809 09:30:02.332175 32310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 09:30:02.334275 32344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:02.335347 32310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 09:30:02.335477 32310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/wal
uuid: "63b152680487497b80946acedb5ae7e4"
format_stamp: "Formatted at 2026-08-09 09:30:02 on dist-test-slave-7vjv"
server_key: "9d79e1cab00a908248f78c6b96af18af"
server_key_iv: "9a73bbb5d4ccaddec77098749defbb3d"
server_key_version: "encryptionkey@0"
I20260809 09:30:02.335584 32310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:02.347235 32310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:02.350108 32310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:02.350286 32310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:02.356488 32310 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.60:41293
I20260809 09:30:02.356513 32428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.60:41293 every 8 connection(s)
I20260809 09:30:02.357276 32310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
I20260809 09:30:02.358372 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 32310
I20260809 09:30:02.358512 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786267646682796-9624-0/minicluster-data/master-2/wal/instance
I20260809 09:30:02.358772 9624 external_mini_cluster.cc:1428] Setting key b753cbe09a20baa862dda641bc853285
I20260809 09:30:02.360005 32429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:02.364068 32429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.60@KRBTEST.COM
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267802, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.60@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:02.372121 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.359953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.9.102.60:41293)
0809 09:30:02.360369 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.360389 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.360428 (+ 39us) server_negotiation.cc:407] Connection header received
0809 09:30:02.361172 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.361193 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.361551 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.361855 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.362422 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.363677 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.364441 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.364839 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.365015 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.367471 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.367478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.367490 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.367517 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.369338 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.369729 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.369738 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.369744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.369881 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.370155 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.370157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.370158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.370516 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.370670 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.370958 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.371216 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":175,"threads_started":1}
I20260809 09:30:02.373889 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.365579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.9.102.62:39769)
0809 09:30:02.365720 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.365724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.366035 (+ 311us) server_negotiation.cc:407] Connection header received
0809 09:30:02.366194 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.366196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.366233 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.366293 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.366970 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.367396 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.368534 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.368715 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.368785 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.370875 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.370878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.370880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.370900 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.372236 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.372734 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.372737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.372738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.372809 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.373088 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.373091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.373093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.373253 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.373358 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.373663 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.373759 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 09:30:02.374131 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.365513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39769 (local address 127.0.0.1:36802)
0809 09:30:02.365896 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.365948 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:02.366101 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:02.366365 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:02.366371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:02.366703 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:02.366853 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.366859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.367502 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.367505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.368435 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.368449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.368819 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.368822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.368918 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.369336 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:02.369350 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:02.370751 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:02.372362 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.372370 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.372375 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.372625 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.372894 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.372899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.372901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.372985 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.373357 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:02.373363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:02.373465 (+ 102us) client_negotiation.cc:769] Sending connection context
0809 09:30:02.373606 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 09:30:02.373761 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":136,"threads_started":1}
I20260809 09:30:02.375236 32429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267802, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.60@KRBTEST.COM for kudu/127.9.102.61@KRBTEST.COM
I20260809 09:30:02.381827 32294 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } attempt: 3
I20260809 09:30:02.383169 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.375794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.9.102.61:38781)
0809 09:30:02.375922 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.375928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.375943 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:30:02.375982 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.375985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.376039 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.376140 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.376477 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.377122 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.377964 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.378246 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.378350 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.380301 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.380304 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.380307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.380333 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.381541 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.381949 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.381951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.381953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.382037 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.382333 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.382338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.382341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.382539 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.382641 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.382887 (+ 246us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.383018 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 09:30:02.383598 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.375684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:38781 (local address 127.0.0.1:47326)
0809 09:30:02.375808 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.375819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:02.375886 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:02.376150 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:02.376152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:02.376292 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:02.376375 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.376380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.377219 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.377221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.377848 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.377856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.378355 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.378358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.378420 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.378827 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:02.378840 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:02.380181 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:02.381634 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.381636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.381637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.381851 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.382131 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.382134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.382135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.382236 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.382630 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:02.382636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:02.382717 (+ 81us) client_negotiation.cc:769] Sending connection context
0809 09:30:02.383311 (+ 594us) client_negotiation.cc:240] Negotiation successful
0809 09:30:02.383456 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260809 09:30:02.384243 32429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:30:02.384498 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.374198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.9.102.62:39769)
0809 09:30:02.374295 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.374300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.374572 (+ 272us) server_negotiation.cc:407] Connection header received
0809 09:30:02.374669 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.374672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.374708 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.374799 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.375517 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.376529 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.377380 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.377654 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.377744 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.379848 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.379851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.379853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.379872 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.381403 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.381821 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.381824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.381826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.381902 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.382230 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.382233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.382235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.382727 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.382958 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.384228 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.384370 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267802, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.61@KRBTEST.COM for kudu/127.9.102.60@KRBTEST.COM
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267802, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.60@KRBTEST.COM for kudu/127.9.102.60@KRBTEST.COM
I20260809 09:30:02.390518 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.382182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58290)
0809 09:30:02.382408 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.382420 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:02.382478 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:02.382678 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:02.382680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:02.382837 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:02.382935 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.382942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.384251 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.384254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.384938 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.384946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.385403 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.385407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.385483 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.385882 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:02.385892 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:02.387202 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:02.388989 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.388992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.388994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.389183 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.389493 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.389496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.389497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.389596 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.390007 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:02.390012 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:02.390115 (+ 103us) client_negotiation.cc:769] Sending connection context
0809 09:30:02.390215 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 09:30:02.390331 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260809 09:30:02.390985 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.382395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.9.102.60:41293)
0809 09:30:02.382479 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.382482 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.382501 (+ 19us) server_negotiation.cc:407] Connection header received
0809 09:30:02.382543 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.382545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.382571 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.382650 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.383038 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.384051 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.385042 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.385273 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.385686 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.387321 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.387324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.387326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.387347 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.388889 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.389316 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.389319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.389320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.389395 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.389676 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.389679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.389680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.389898 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.389987 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.390732 (+ 745us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.390849 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":25}
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.61@KRBTEST.COM
I20260809 09:30:02.393883 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.384667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58298)
0809 09:30:02.384769 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.384781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:02.384842 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:02.385318 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:02.385320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:02.385465 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:02.385514 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.385518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.386408 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.386411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.387071 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.387082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.387655 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.387661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.387760 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.388183 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:02.388196 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:02.389521 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:02.392466 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.392468 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.392470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.392683 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.392944 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.392947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.392949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.393024 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.393437 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:02.393440 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:02.393517 (+ 77us) client_negotiation.cc:769] Sending connection context
0809 09:30:02.393623 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 09:30:02.393744 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 09:30:02.393935 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.384848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.9.102.60:41293)
0809 09:30:02.385141 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.385154 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.385164 (+ 10us) server_negotiation.cc:407] Connection header received
0809 09:30:02.385194 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.385196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.385231 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.385310 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.385619 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.386299 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.387197 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.387515 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.387624 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.389640 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.389646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.389650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.389674 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.392378 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.392777 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.392779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.392780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.392848 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.393143 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.393147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.393149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.393334 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.393443 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.393675 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.393797 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":234,"thread_start_us":98,"threads_started":1}
I20260809 09:30:02.394172 32294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6: Bootstrap starting.
I20260809 09:30:02.394800 32172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } attempt: 3
I20260809 09:30:02.396003 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.386446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.9.102.61:38781)
0809 09:30:02.386554 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.386558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.386569 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:30:02.386599 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.386602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.386636 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.386719 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.387363 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.388263 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.389165 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.389452 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.389854 (+ 402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.391951 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.391954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.391956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.391981 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.393823 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.394495 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.394498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.394499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.394583 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.395014 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.395017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.395018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.395343 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.395568 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.395759 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.395853 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 09:30:02.397893 32294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:02.398300 32429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4: Bootstrap starting.
I20260809 09:30:02.398783 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Aug 09 09:30:02 dist-test-slave-7vjv krb5kdc[32034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267801, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.60@KRBTEST.COM
I20260809 09:30:02.400792 32294 log.cc:824] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:02.401109 32429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:02.402204 32429 log.cc:824] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:02.403806 32294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6: No bootstrap required, opened a new log
I20260809 09:30:02.404502 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.395155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:41293 (local address 127.0.0.1:58300)
0809 09:30:02.395240 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:02.395254 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:02.395331 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:02.395550 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:02.395553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:02.395743 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:02.395843 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.395851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.396944 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.396947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.397656 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:02.397666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.398234 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.398237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:02.398325 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.398732 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:02.398745 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:02.400162 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:02.402890 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.402892 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.402893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.403164 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.403464 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:02.403467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:02.403469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:02.403550 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:02.403992 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:02.403995 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:02.404113 (+ 118us) client_negotiation.cc:769] Sending connection context
0809 09:30:02.404200 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 09:30:02.404331 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260809 09:30:02.405543 32441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.395190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.9.102.60:41293)
0809 09:30:02.395315 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.395319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.395330 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:30:02.395409 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.395411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.395441 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.395951 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.396035 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.396819 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.397787 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.398112 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.398553 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.400291 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.400294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.400296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.400318 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.402540 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.403284 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.403287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.403289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.403377 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.403670 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.403673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.403675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.403837 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.405209 (+ 1372us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.405280 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.405398 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 09:30:02.408017 32429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4: No bootstrap required, opened a new log
I20260809 09:30:02.408950 32172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15: Bootstrap starting.
I20260809 09:30:02.409300 32294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } }
I20260809 09:30:02.409518 32294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:02.409584 32294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30cbc3b5ef984e19a4087f4515feb8e6, State: Initialized, Role: FOLLOWER
I20260809 09:30:02.411264 32172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:02.411301 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.401155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.9.102.60:41293)
0809 09:30:02.401247 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.401252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.402513 (+ 1261us) server_negotiation.cc:407] Connection header received
0809 09:30:02.402695 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.402697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.402731 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.402795 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.403707 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.404347 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.405577 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.405922 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.405997 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.407622 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.407628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.407633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.407660 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.409358 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.410025 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.410029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.410031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.410116 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.410394 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.410397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.410398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.410591 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.410688 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.411048 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.411151 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 09:30:02.411897 32294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } }
I20260809 09:30:02.412294 32429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } }
I20260809 09:30:02.412509 32429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:02.412598 32429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b152680487497b80946acedb5ae7e4, State: Initialized, Role: FOLLOWER
I20260809 09:30:02.413044 32429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } }
I20260809 09:30:02.413482 32172 log.cc:824] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:02.413578 32294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:30:02.413501 32457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } } }
I20260809 09:30:02.413734 32457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:30:02.414180 32429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 09:30:02.419426 32498 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:30:02.419524 32498 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260809 09:30:02.419723 32484 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:30:02.419872 32484 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:30:02.419916 32455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } } }
I20260809 09:30:02.420048 32455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:30:02.417469 32184 master_runner.cc:186] Error getting master registration for 127.9.102.62:39769: OK, instance_id { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" instance_seqno: 1786267801934713 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 09:30:02.421720 32184 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 09:30:02.422433 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.401690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.9.102.62:39769)
0809 09:30:02.401786 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.401789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.405117 (+ 3328us) server_negotiation.cc:407] Connection header received
0809 09:30:02.408199 (+ 3082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.408205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.408310 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.408401 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.413896 (+ 5495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.414376 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.415353 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.415660 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.415752 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.418269 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.418272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.418275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.418297 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.419884 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.420407 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.420410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.420412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.420498 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.420812 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.420816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.420817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.421015 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.421126 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.421840 (+ 714us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.422267 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 09:30:02.420579 32310 master_runner.cc:186] Error getting master registration for 127.9.102.62:39769: OK, instance_id { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" instance_seqno: 1786267801934713 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 09:30:02.423738 32310 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 09:30:02.426028 32172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15: No bootstrap required, opened a new log
I20260809 09:30:02.429381 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:02.400811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.9.102.61:38781)
0809 09:30:02.400896 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:02.400900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:02.401626 (+ 726us) server_negotiation.cc:407] Connection header received
0809 09:30:02.401819 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:02.401823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:02.401862 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:02.401911 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:02.405264 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.406218 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.410771 (+ 4553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:02.411077 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:02.411153 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:02.421448 (+ 10295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:02.421451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:02.421454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:02.421475 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:02.423142 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.424542 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.424546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.424547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.424631 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:02.428498 (+ 3867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:02.428501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:02.428503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:02.428703 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:02.428842 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:02.429123 (+ 281us) server_negotiation.cc:299] Negotiation successful
0809 09:30:02.429233 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 09:30:02.430321 32172 raft_consensus.cc:348] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } }
I20260809 09:30:02.430546 32172 raft_consensus.cc:374] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:02.430593 32172 raft_consensus.cc:729] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3532e0130242dbad17f5ef81f0ef15, State: Initialized, Role: FOLLOWER
I20260809 09:30:02.431097 32172 consensus_queue.cc:260] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } }
I20260809 09:30:02.431617 32501 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } } }
I20260809 09:30:02.433104 32172 sys_catalog.cc:564] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:30:02.432057 32501 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 09:30:02.438143 32520 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:30:02.438264 32520 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:30:02.442431 32058 master_runner.cc:263] No leader master found from master da3532e0130242dbad17f5ef81f0ef15
I20260809 09:30:02.442502 32058 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 09:30:02.601500 32355 tablet.cc:2369] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:30:02.631552 32213 tablet.cc:2369] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:30:02.682205 32091 tablet.cc:2369] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:30:02.682684 32501 raft_consensus.cc:482] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:30:02.682806 32501 raft_consensus.cc:504] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } }
I20260809 09:30:02.683517 32501 leader_election.cc:288] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30cbc3b5ef984e19a4087f4515feb8e6 (127.9.102.61:38781), 63b152680487497b80946acedb5ae7e4 (127.9.102.60:41293)
I20260809 09:30:02.684018 32392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da3532e0130242dbad17f5ef81f0ef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b152680487497b80946acedb5ae7e4" is_pre_election: true
I20260809 09:30:02.684002 32244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da3532e0130242dbad17f5ef81f0ef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" is_pre_election: true
I20260809 09:30:02.684322 32244 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3532e0130242dbad17f5ef81f0ef15 in term 0.
I20260809 09:30:02.684322 32392 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3532e0130242dbad17f5ef81f0ef15 in term 0.
I20260809 09:30:02.684689 32088 leader_election.cc:302] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b152680487497b80946acedb5ae7e4, da3532e0130242dbad17f5ef81f0ef15; no voters:
I20260809 09:30:02.684914 32501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:30:02.685038 32501 raft_consensus.cc:482] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:30:02.685096 32501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:02.686110 32501 raft_consensus.cc:504] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } }
I20260809 09:30:02.686419 32501 leader_election.cc:288] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [CANDIDATE]: Term 1 election: Requested vote from peers 30cbc3b5ef984e19a4087f4515feb8e6 (127.9.102.61:38781), 63b152680487497b80946acedb5ae7e4 (127.9.102.60:41293)
I20260809 09:30:02.686805 32392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da3532e0130242dbad17f5ef81f0ef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b152680487497b80946acedb5ae7e4"
I20260809 09:30:02.686800 32244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "da3532e0130242dbad17f5ef81f0ef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30cbc3b5ef984e19a4087f4515feb8e6"
I20260809 09:30:02.686920 32244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:02.686921 32392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:02.688122 32392 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3532e0130242dbad17f5ef81f0ef15 in term 1.
I20260809 09:30:02.688122 32244 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3532e0130242dbad17f5ef81f0ef15 in term 1.
I20260809 09:30:02.688396 32088 leader_election.cc:302] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63b152680487497b80946acedb5ae7e4, da3532e0130242dbad17f5ef81f0ef15; no voters:
I20260809 09:30:02.688582 32501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:30:02.688776 32501 raft_consensus.cc:686] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [term 1 LEADER]: Becoming Leader. State: Replica: da3532e0130242dbad17f5ef81f0ef15, State: Running, Role: LEADER
I20260809 09:30:02.689085 32501 consensus_queue.cc:237] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } }
I20260809 09:30:02.690850 32523 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da3532e0130242dbad17f5ef81f0ef15. Latest consensus state: current_term: 1 leader_uuid: "da3532e0130242dbad17f5ef81f0ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } } }
I20260809 09:30:02.690956 32523 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:02.691332 32530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:30:02.692217 32530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:30:02.696473 32392 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [term 1 FOLLOWER]: Refusing update from remote peer da3532e0130242dbad17f5ef81f0ef15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:30:02.696506 32244 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [term 1 FOLLOWER]: Refusing update from remote peer da3532e0130242dbad17f5ef81f0ef15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:30:02.697049 32523 consensus_queue.cc:1035] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:30:02.697185 32501 consensus_queue.cc:1035] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:30:02.701185 32455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da3532e0130242dbad17f5ef81f0ef15. Latest consensus state: current_term: 1 leader_uuid: "da3532e0130242dbad17f5ef81f0ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } } }
I20260809 09:30:02.701186 32457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da3532e0130242dbad17f5ef81f0ef15. Latest consensus state: current_term: 1 leader_uuid: "da3532e0130242dbad17f5ef81f0ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } } }
I20260809 09:30:02.701293 32457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:30:02.701295 32455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:30:02.703689 32539 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da3532e0130242dbad17f5ef81f0ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } } }
I20260809 09:30:02.703794 32539 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:02.703884 32539 sys_catalog.cc:455] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "da3532e0130242dbad17f5ef81f0ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } } }
I20260809 09:30:02.703940 32539 sys_catalog.cc:458] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:02.704221 32457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da3532e0130242dbad17f5ef81f0ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } } }
I20260809 09:30:02.704309 32457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:30:02.704363 32455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "da3532e0130242dbad17f5ef81f0ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da3532e0130242dbad17f5ef81f0ef15" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39769 } } peers { permanent_uuid: "30cbc3b5ef984e19a4087f4515feb8e6" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 38781 } } peers { permanent_uuid: "63b152680487497b80946acedb5ae7e4" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 41293 } } }
I20260809 09:30:02.704463 32455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:30:02.704664 32534 mvcc.cc:205] Tried to move back new op lower bound from 7316552919839203328 to 7316552919816794112. Current Snapshot: MvccSnapshot[applied={T|T < 7316552919839203328}]
I20260809 09:30:02.704668 32533 mvcc.cc:205] Tried to move back new op lower bound from 7316552919839203328 to 7316552919816794112. Current Snapshot: MvccSnapshot[applied={T|T < 7316552919839203328}]
I20260809 09:30:02.705186 32530 catalog_manager.cc:1347] Generated new cluster ID: c53a06dec32b4f8e84f245ef26efd9f4
I20260809 09:30:02.705250 32530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:30:02.712455 32530 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:30:02.713574 32530 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:30:02.723347 32530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da3532e0130242dbad17f5ef81f0ef15: Generated new TSK 0
I20260809 09:30:02.724176 32530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:30:03.420725 32498 catalog_manager.cc:1259] Loaded cluster ID: c53a06dec32b4f8e84f245ef26efd9f4
I20260809 09:30:03.420806 32498 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4: loading cluster ID for follower catalog manager: success
I20260809 09:30:03.421041 32484 catalog_manager.cc:1259] Loaded cluster ID: c53a06dec32b4f8e84f245ef26efd9f4
I20260809 09:30:03.421116 32484 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6: loading cluster ID for follower catalog manager: success
I20260809 09:30:03.422345 32498 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4: acquiring CA information for follower catalog manager: success
I20260809 09:30:03.422585 32484 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6: acquiring CA information for follower catalog manager: success
I20260809 09:30:03.423727 32498 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 63b152680487497b80946acedb5ae7e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 09:30:03.423812 32484 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30cbc3b5ef984e19a4087f4515feb8e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 09:30:03.790692 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 32058
I20260809 09:30:03.804276 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 32184
I20260809 09:30:03.818040 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 32310
2026-08-09T09:30:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3340 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:30:03 dist-test-slave-7vjv krb5kdc[32559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:30:03 dist-test-slave-7vjv krb5kdc[32559](info): set up 2 sockets
Aug 09 09:30:03 dist-test-slave-7vjv krb5kdc[32559](info): commencing operation
krb5kdc: starting...
W20260809 09:30:04.989300 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:30:05 dist-test-slave-7vjv krb5kdc[32559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:30:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:30:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:30:05.098853 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:33903
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:33903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:05.197463 32579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:05.197697 32579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:05.197758 32579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:05.201407 32579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:30:05.201481 32579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:05.201519 32579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:05.201586 32579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:30:05.201617 32579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:30:05.206032 32579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:33903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:33903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:05.206801 32579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:05.207643 32579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:05.214852 32589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:05.214844 32584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:05.215121 32587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:05.215364 32579 server_base.cc:1034] running on GCE node
I20260809 09:30:05.215837 32579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:05.216878 32579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:05.218111 32579 hybrid_clock.cc:648] HybridClock initialized: now 1786267805218094 us; error 43 us; skew 500 ppm
Aug 09 09:30:05 dist-test-slave-7vjv krb5kdc[32559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267805, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:05.220618 32579 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:30:05.221335 32579 webserver.cc:460] Webserver started at http://127.9.102.62:46773/ using document root <none> and password file <none>
I20260809 09:30:05.221791 32579 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:05.221848 32579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:05.221992 32579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:30:05.223723 32579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "a153c048cdac42e19bbb479a4d499683"
format_stamp: "Formatted at 2026-08-09 09:30:05 on dist-test-slave-7vjv"
server_key: "fa5d09f70953983dec6de1700fb0620b"
server_key_iv: "d37e8028d30df292f90b07f9912b67f0"
server_key_version: "encryptionkey@0"
I20260809 09:30:05.224260 32579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "a153c048cdac42e19bbb479a4d499683"
format_stamp: "Formatted at 2026-08-09 09:30:05 on dist-test-slave-7vjv"
server_key: "fa5d09f70953983dec6de1700fb0620b"
server_key_iv: "d37e8028d30df292f90b07f9912b67f0"
server_key_version: "encryptionkey@0"
I20260809 09:30:05.227265 32579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 09:30:05.229328 32599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:05.230409 32579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:30:05.230505 32579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "a153c048cdac42e19bbb479a4d499683"
format_stamp: "Formatted at 2026-08-09 09:30:05 on dist-test-slave-7vjv"
server_key: "fa5d09f70953983dec6de1700fb0620b"
server_key_iv: "d37e8028d30df292f90b07f9912b67f0"
server_key_version: "encryptionkey@0"
I20260809 09:30:05.230593 32579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:05.240139 32579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:05.242489 32579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:05.242626 32579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:05.248836 32579 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:33903
I20260809 09:30:05.248867 32690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:33903 every 8 connection(s)
I20260809 09:30:05.249718 32579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:30:05.252243 32691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:05.254065 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 32579
I20260809 09:30:05.254154 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:30:05.254416 9624 external_mini_cluster.cc:1428] Setting key d07723dd2379b217c647cb5a259a4821
I20260809 09:30:05.258692 32691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683: Bootstrap starting.
I20260809 09:30:05.261212 32691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:05.262076 32691 log.cc:824] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683: Log is configured to *not* fsync() on all Append() calls
Aug 09 09:30:05 dist-test-slave-7vjv krb5kdc[32559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:05.263935 32691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683: No bootstrap required, opened a new log
I20260809 09:30:05.268065 32691 raft_consensus.cc:348] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a153c048cdac42e19bbb479a4d499683" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33903 } }
I20260809 09:30:05.268290 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:05.255639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.9.102.62:33903)
0809 09:30:05.256026 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:05.256045 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:05.256085 (+ 40us) server_negotiation.cc:407] Connection header received
0809 09:30:05.256830 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:05.256853 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:05.257275 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:05.257631 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:05.258290 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:05.259729 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:05.260595 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:05.261251 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:05.261409 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:05.263860 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:05.263866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:05.263880 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:05.263906 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:05.265666 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:05.266035 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:05.266041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:05.266049 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:05.266223 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:05.266464 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:05.266467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:05.266468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:05.266890 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:05.267079 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:05.267232 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 09:30:05.267485 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":91,"threads_started":1}
I20260809 09:30:05.268308 32691 raft_consensus.cc:374] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:05.268350 32691 raft_consensus.cc:729] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a153c048cdac42e19bbb479a4d499683, State: Initialized, Role: FOLLOWER
I20260809 09:30:05.268710 32691 consensus_queue.cc:260] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a153c048cdac42e19bbb479a4d499683" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33903 } }
I20260809 09:30:05.268850 32691 raft_consensus.cc:388] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:30:05.268929 32691 raft_consensus.cc:482] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:30:05.269042 32691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:05.269930 32691 raft_consensus.cc:504] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a153c048cdac42e19bbb479a4d499683" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33903 } }
I20260809 09:30:05.270260 32691 leader_election.cc:302] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a153c048cdac42e19bbb479a4d499683; no voters:
I20260809 09:30:05.270584 32691 leader_election.cc:288] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:30:05.270668 32697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:30:05.270885 32697 raft_consensus.cc:686] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [term 1 LEADER]: Becoming Leader. State: Replica: a153c048cdac42e19bbb479a4d499683, State: Running, Role: LEADER
I20260809 09:30:05.270948 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:30:05.271207 32697 consensus_queue.cc:237] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a153c048cdac42e19bbb479a4d499683" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33903 } }
I20260809 09:30:05.271641 32691 sys_catalog.cc:564] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:30:05.272867 32699 sys_catalog.cc:455] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a153c048cdac42e19bbb479a4d499683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a153c048cdac42e19bbb479a4d499683" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33903 } } }
I20260809 09:30:05.272987 32699 sys_catalog.cc:458] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:05.273238 32700 sys_catalog.cc:455] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a153c048cdac42e19bbb479a4d499683. Latest consensus state: current_term: 1 leader_uuid: "a153c048cdac42e19bbb479a4d499683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a153c048cdac42e19bbb479a4d499683" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 33903 } } }
I20260809 09:30:05.273334 32700 sys_catalog.cc:458] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683 [sys.catalog]: This master's current role is: LEADER
W20260809 09:30:05.275888 32726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:30:05.275974 32726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:30:05.276069 32718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:30:05.276743 32718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:30:05.281656 32718 catalog_manager.cc:1347] Generated new cluster ID: aec241155a6f48ef8c9eee76bf57a9f7
I20260809 09:30:05.281711 32718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:30:05.293260 32718 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:30:05.294399 32718 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:30:05.306998 32718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a153c048cdac42e19bbb479a4d499683: Generated new TSK 0
I20260809 09:30:05.307554 32718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:30:05.326277 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 32579
2026-08-09T09:30:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1513 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:30:05 dist-test-slave-7vjv krb5kdc[32739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:30:05 dist-test-slave-7vjv krb5kdc[32739](info): set up 2 sockets
Aug 09 09:30:05 dist-test-slave-7vjv krb5kdc[32739](info): commencing operation
krb5kdc: starting...
W20260809 09:30:06.509509 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:30:06 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:30:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:30:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:30:06.618682 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40807
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:40807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:06.715307 32763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:06.715528 32763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:06.715564 32763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:06.719529 32763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:30:06.719578 32763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:06.719607 32763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:06.719621 32763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:30:06.719635 32763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:30:06.723976 32763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:40807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:06.724748 32763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:06.725528 32763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:06.732784 304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:06.732776 308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:06.732910 303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:06.733170 32763 server_base.cc:1034] running on GCE node
I20260809 09:30:06.733646 32763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:06.734508 32763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:06.735697 32763 hybrid_clock.cc:648] HybridClock initialized: now 1786267806735672 us; error 46 us; skew 500 ppm
Aug 09 09:30:06 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267806, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:06.738214 32763 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:30:06.739027 32763 webserver.cc:460] Webserver started at http://127.9.102.62:35297/ using document root <none> and password file <none>
I20260809 09:30:06.739483 32763 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:06.739574 32763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:06.739787 32763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:30:06.741508 32763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "d77cb80bc89c4cd2baea34c10760dc76"
format_stamp: "Formatted at 2026-08-09 09:30:06 on dist-test-slave-7vjv"
server_key: "1de7dfecd8991203b253d0de0b3b2755"
server_key_iv: "3f46d4e2f1eee7d06658656552b924dc"
server_key_version: "encryptionkey@0"
I20260809 09:30:06.742038 32763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "d77cb80bc89c4cd2baea34c10760dc76"
format_stamp: "Formatted at 2026-08-09 09:30:06 on dist-test-slave-7vjv"
server_key: "1de7dfecd8991203b253d0de0b3b2755"
server_key_iv: "3f46d4e2f1eee7d06658656552b924dc"
server_key_version: "encryptionkey@0"
I20260809 09:30:06.745298 32763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260809 09:30:06.747429 324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:06.748494 32763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:06.748616 32763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "d77cb80bc89c4cd2baea34c10760dc76"
format_stamp: "Formatted at 2026-08-09 09:30:06 on dist-test-slave-7vjv"
server_key: "1de7dfecd8991203b253d0de0b3b2755"
server_key_iv: "3f46d4e2f1eee7d06658656552b924dc"
server_key_version: "encryptionkey@0"
I20260809 09:30:06.748708 32763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:06.759801 32763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:06.762137 32763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:06.762311 32763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:06.768610 32763 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:40807
I20260809 09:30:06.768841 405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:40807 every 8 connection(s)
I20260809 09:30:06.769385 32763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:30:06.771726 407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:06.773882 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 32763
I20260809 09:30:06.773975 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:30:06.774223 9624 external_mini_cluster.cc:1428] Setting key 37cdf5c6f2b338299879faf421110d7f
I20260809 09:30:06.776962 407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Bootstrap starting.
I20260809 09:30:06.779353 407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:06.780190 407 log.cc:824] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:06.782390 407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: No bootstrap required, opened a new log
Aug 09 09:30:06 dist-test-slave-7vjv krb5kdc[32739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:06.785483 407 raft_consensus.cc:348] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } }
I20260809 09:30:06.785701 407 raft_consensus.cc:374] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:06.785779 407 raft_consensus.cc:729] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77cb80bc89c4cd2baea34c10760dc76, State: Initialized, Role: FOLLOWER
I20260809 09:30:06.786237 407 consensus_queue.cc:260] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } }
I20260809 09:30:06.786408 407 raft_consensus.cc:388] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:30:06.786473 407 raft_consensus.cc:482] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:30:06.786569 407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:06.787458 407 raft_consensus.cc:504] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } }
I20260809 09:30:06.787763 407 leader_election.cc:302] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d77cb80bc89c4cd2baea34c10760dc76; no voters:
I20260809 09:30:06.788055 407 leader_election.cc:288] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:30:06.788206 422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:30:06.788331 415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:06.775486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.9.102.62:40807)
0809 09:30:06.775887 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:06.775900 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:06.775930 (+ 30us) server_negotiation.cc:407] Connection header received
0809 09:30:06.776516 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:06.776538 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:06.776812 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:06.777047 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:06.777889 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:06.778866 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:06.779959 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:06.780609 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:06.780780 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:06.783332 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:06.783336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:06.783347 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:06.783370 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:06.785447 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:06.785922 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:06.785927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:06.785931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:06.786062 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:06.786315 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:06.786318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:06.786319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:06.786795 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:06.786973 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:06.787183 (+ 210us) server_negotiation.cc:299] Negotiation successful
0809 09:30:06.787459 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":170,"threads_started":1}
I20260809 09:30:06.788421 422 raft_consensus.cc:686] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 1 LEADER]: Becoming Leader. State: Replica: d77cb80bc89c4cd2baea34c10760dc76, State: Running, Role: LEADER
I20260809 09:30:06.788714 422 consensus_queue.cc:237] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } }
I20260809 09:30:06.789171 407 sys_catalog.cc:564] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:30:06.789732 424 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d77cb80bc89c4cd2baea34c10760dc76. Latest consensus state: current_term: 1 leader_uuid: "d77cb80bc89c4cd2baea34c10760dc76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } } }
I20260809 09:30:06.790486 424 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:06.790352 427 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d77cb80bc89c4cd2baea34c10760dc76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } } }
I20260809 09:30:06.790817 427 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:06.790896 438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:30:06.793987 438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:30:06.798954 438 catalog_manager.cc:1347] Generated new cluster ID: d67cfc51a1c64a27872e89584dfb15de
I20260809 09:30:06.799018 438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:30:06.805711 438 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:30:06.806418 438 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:30:06.812005 438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Generated new TSK 0
I20260809 09:30:06.812517 438 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:30:06.840906 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40807
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:06.941972 456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:06.942210 456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:06.942272 456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:06.945806 456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:06.945880 456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:06.945963 456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:30:06.950317 456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:06.951171 456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:06.952075 456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:06.961035 464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:06.961053 467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:06.961078 456 server_base.cc:1034] running on GCE node
W20260809 09:30:06.961046 469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:06.961588 456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:06.962126 456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:06.963295 456 hybrid_clock.cc:648] HybridClock initialized: now 1786267806963288 us; error 25 us; skew 500 ppm
Aug 09 09:30:06 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267806, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:06.965921 456 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:30:06.966746 456 webserver.cc:460] Webserver started at http://127.9.102.1:38097/ using document root <none> and password file <none>
I20260809 09:30:06.967217 456 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:06.967272 456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:06.967490 456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:30:06.969173 456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "a7ba480de48747e99a964e8f9a50f440"
format_stamp: "Formatted at 2026-08-09 09:30:06 on dist-test-slave-7vjv"
server_key: "61cb33e56884fc6d8b350d2b008fccdb"
server_key_iv: "c6e541cb9e5b884c94c7cc4bb2390427"
server_key_version: "encryptionkey@0"
I20260809 09:30:06.969648 456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "a7ba480de48747e99a964e8f9a50f440"
format_stamp: "Formatted at 2026-08-09 09:30:06 on dist-test-slave-7vjv"
server_key: "61cb33e56884fc6d8b350d2b008fccdb"
server_key_iv: "c6e541cb9e5b884c94c7cc4bb2390427"
server_key_version: "encryptionkey@0"
I20260809 09:30:06.972937 456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:30:06.974964 480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:06.976028 456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:06.976183 456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "a7ba480de48747e99a964e8f9a50f440"
format_stamp: "Formatted at 2026-08-09 09:30:06 on dist-test-slave-7vjv"
server_key: "61cb33e56884fc6d8b350d2b008fccdb"
server_key_iv: "c6e541cb9e5b884c94c7cc4bb2390427"
server_key_version: "encryptionkey@0"
I20260809 09:30:06.976297 456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:06.991904 456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:06.994359 456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:06.994505 456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:06.994920 456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:06.995594 456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:30:06.995636 456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:06.995669 456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:30:06.995697 456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:07.003773 456 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:43639
I20260809 09:30:07.003814 678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:43639 every 8 connection(s)
I20260809 09:30:07.004563 456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:30:07.006309 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 456
I20260809 09:30:07.006459 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:30:07.006774 9624 external_mini_cluster.cc:1428] Setting key 4be119cf42aed647a11f27012aa5e6f1
Aug 09 09:30:07 dist-test-slave-7vjv krb5kdc[32739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267806, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:30:07.017742 415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.006121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:41763 (local address 127.9.102.62:40807)
0809 09:30:07.006257 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:07.006260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:07.007139 (+ 879us) server_negotiation.cc:407] Connection header received
0809 09:30:07.008626 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:07.008629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:07.008679 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:07.008753 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:07.009804 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.010440 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.011495 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.011843 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.011958 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.014145 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:07.014151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:07.014156 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:07.014188 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:07.015728 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:07.016246 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:07.016250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:07.016252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:07.016339 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:07.016679 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:07.016685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:07.016688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:07.016940 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:07.017048 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:07.017492 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 09:30:07.017632 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 09:30:07.018399 682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.006509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:41763)
0809 09:30:07.006989 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:07.007028 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:07.008435 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:07.008908 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:07.008914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:07.009232 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:07.009667 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.009679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.010566 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.010569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.011365 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.011374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.011975 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.011977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.012124 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.012641 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:07.012655 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:07.013969 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:07.015840 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:07.015845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:07.015854 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:07.016087 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:07.016434 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:07.016436 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:07.016437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:07.016571 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:07.017045 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:07.017049 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:07.017235 (+ 186us) client_negotiation.cc:769] Sending connection context
0809 09:30:07.017405 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 09:30:07.017633 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":139,"threads_started":1}
I20260809 09:30:07.019517 679 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40807
I20260809 09:30:07.019740 679 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
I20260809 09:30:07.020300 679 heartbeater.cc:511] Master 127.9.102.62:40807 requested a full tablet report, sending...
I20260809 09:30:07.021708 350 ts_manager.cc:194] Registered new tserver with Master: a7ba480de48747e99a964e8f9a50f440 (127.9.102.1:43639)
I20260809 09:30:07.022843 350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:41763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:30:07.034307 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40807
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:07.132529 692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:07.132774 692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:07.132813 692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:07.136463 692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:07.136504 692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:07.136557 692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:30:07.141042 692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:07.141750 692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:07.142480 692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:07.150588 706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:07.150650 700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:07.150588 701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:07.150770 692 server_base.cc:1034] running on GCE node
I20260809 09:30:07.151202 692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:07.151775 692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:07.152949 692 hybrid_clock.cc:648] HybridClock initialized: now 1786267807152936 us; error 27 us; skew 500 ppm
Aug 09 09:30:07 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267807, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:07.160499 692 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:30:07.161276 692 webserver.cc:460] Webserver started at http://127.9.102.2:34289/ using document root <none> and password file <none>
I20260809 09:30:07.161753 692 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:07.161840 692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:07.162047 692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:30:07.163766 692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "9f0e48d0c8e246aea9a13ddd0b972217"
format_stamp: "Formatted at 2026-08-09 09:30:07 on dist-test-slave-7vjv"
server_key: "5dddd06c6c393a9c1a1d2ca956cd2be1"
server_key_iv: "79b8d7ce1a6d8f80133c42f316f69d01"
server_key_version: "encryptionkey@0"
I20260809 09:30:07.164328 692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "9f0e48d0c8e246aea9a13ddd0b972217"
format_stamp: "Formatted at 2026-08-09 09:30:07 on dist-test-slave-7vjv"
server_key: "5dddd06c6c393a9c1a1d2ca956cd2be1"
server_key_iv: "79b8d7ce1a6d8f80133c42f316f69d01"
server_key_version: "encryptionkey@0"
I20260809 09:30:07.167562 692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 09:30:07.169620 723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:07.170635 692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:30:07.170768 692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "9f0e48d0c8e246aea9a13ddd0b972217"
format_stamp: "Formatted at 2026-08-09 09:30:07 on dist-test-slave-7vjv"
server_key: "5dddd06c6c393a9c1a1d2ca956cd2be1"
server_key_iv: "79b8d7ce1a6d8f80133c42f316f69d01"
server_key_version: "encryptionkey@0"
I20260809 09:30:07.170868 692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:07.187520 692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:07.190284 692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:07.190451 692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:07.190966 692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:07.191674 692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:30:07.191743 692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:07.191802 692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:30:07.191849 692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:07.199961 692 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:39087
I20260809 09:30:07.199986 931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:39087 every 8 connection(s)
I20260809 09:30:07.200796 692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
Aug 09 09:30:07 dist-test-slave-7vjv krb5kdc[32739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267807, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:07.210440 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 692
I20260809 09:30:07.210583 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:30:07.210850 9624 external_mini_cluster.cc:1428] Setting key 77f7fa46461310b6303706837ce701cb
I20260809 09:30:07.213501 415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.202327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:53135 (local address 127.9.102.62:40807)
0809 09:30:07.202454 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:07.202458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:07.203564 (+ 1106us) server_negotiation.cc:407] Connection header received
0809 09:30:07.204833 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:07.204835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:07.204880 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:07.204949 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:07.205951 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.206455 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.207413 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.207610 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.207679 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.209913 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:07.209916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:07.209918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:07.209940 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:07.211508 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:07.211975 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:07.211979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:07.211981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:07.212067 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:07.212436 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:07.212439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:07.212441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:07.212653 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:07.212758 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:07.213168 (+ 410us) server_negotiation.cc:299] Negotiation successful
0809 09:30:07.213294 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 09:30:07.214004 934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.202661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:53135)
0809 09:30:07.203434 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:07.203473 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:07.204629 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:07.205119 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:07.205124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:07.205425 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:07.205820 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.205831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.206567 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.206569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.207314 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.207323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.207723 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.207725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.207866 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.208431 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:07.208446 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:07.209769 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:07.211616 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:07.211622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:07.211634 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:07.211875 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:07.212182 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:07.212187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:07.212190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:07.212337 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:07.212755 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:07.212759 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:07.212943 (+ 184us) client_negotiation.cc:769] Sending connection context
0809 09:30:07.213084 (+ 141us) client_negotiation.cc:240] Negotiation successful
0809 09:30:07.213293 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":470,"threads_started":1}
I20260809 09:30:07.215034 932 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40807
I20260809 09:30:07.215255 932 heartbeater.cc:464] Registering TS with master...
I20260809 09:30:07.215801 932 heartbeater.cc:511] Master 127.9.102.62:40807 requested a full tablet report, sending...
I20260809 09:30:07.216828 350 ts_manager.cc:194] Registered new tserver with Master: 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087)
I20260809 09:30:07.217338 350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:53135
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:30:07.236879 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40807
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:07.336299 940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:07.336527 940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:07.336606 940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:07.340073 940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:07.340173 940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:07.340276 940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:30:07.344731 940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:07.345585 940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:07.346406 940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:07.354609 950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:07.354612 953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:07.354595 949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:07.354843 940 server_base.cc:1034] running on GCE node
I20260809 09:30:07.355242 940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:07.355813 940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:07.356961 940 hybrid_clock.cc:648] HybridClock initialized: now 1786267807356951 us; error 25 us; skew 500 ppm
Aug 09 09:30:07 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267807, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:07.359282 940 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:30:07.360091 940 webserver.cc:460] Webserver started at http://127.9.102.3:38765/ using document root <none> and password file <none>
I20260809 09:30:07.360647 940 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:07.360736 940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:07.360939 940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:30:07.362576 940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "4f72ef30dac5468493c7e47c28e0db1c"
format_stamp: "Formatted at 2026-08-09 09:30:07 on dist-test-slave-7vjv"
server_key: "bb5277e2a448d063d3cd0074b6c18b71"
server_key_iv: "c9003a26966708592c358093f97183f7"
server_key_version: "encryptionkey@0"
I20260809 09:30:07.363055 940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "4f72ef30dac5468493c7e47c28e0db1c"
format_stamp: "Formatted at 2026-08-09 09:30:07 on dist-test-slave-7vjv"
server_key: "bb5277e2a448d063d3cd0074b6c18b71"
server_key_iv: "c9003a26966708592c358093f97183f7"
server_key_version: "encryptionkey@0"
I20260809 09:30:07.366219 940 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:30:07.368222 962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:07.369184 940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:07.369313 940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "4f72ef30dac5468493c7e47c28e0db1c"
format_stamp: "Formatted at 2026-08-09 09:30:07 on dist-test-slave-7vjv"
server_key: "bb5277e2a448d063d3cd0074b6c18b71"
server_key_iv: "c9003a26966708592c358093f97183f7"
server_key_version: "encryptionkey@0"
I20260809 09:30:07.369410 940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:07.387924 940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:07.390551 940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:07.390740 940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:07.391252 940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:07.392050 940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:30:07.392139 940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:07.392230 940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:30:07.392282 940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:07.400635 940 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:41373
I20260809 09:30:07.400669 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:41373 every 8 connection(s)
I20260809 09:30:07.401386 940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:30:07.402148 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 940
I20260809 09:30:07.402266 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:30:07.402508 9624 external_mini_cluster.cc:1428] Setting key 91785dc88e62fa49f9e72a5e9ceba15b
Aug 09 09:30:07 dist-test-slave-7vjv krb5kdc[32739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267807, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:07.413815 415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.403133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:57969 (local address 127.9.102.62:40807)
0809 09:30:07.403258 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:07.403262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:07.404239 (+ 977us) server_negotiation.cc:407] Connection header received
0809 09:30:07.405611 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:07.405615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:07.405668 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:07.405745 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:07.406779 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.407267 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.408351 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.408565 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.408670 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.410748 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:07.410758 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:07.410760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:07.410781 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:07.411916 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:07.412436 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:07.412440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:07.412442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:07.412517 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:07.412848 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:07.412851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:07.412852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:07.413008 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:07.413122 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:07.413567 (+ 445us) server_negotiation.cc:299] Negotiation successful
0809 09:30:07.413671 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:30:07.414408 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.403533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:57969)
0809 09:30:07.404062 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:07.404142 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:07.405418 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:07.405906 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:07.405913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:07.406228 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:07.406648 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.406660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.407384 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.407388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.408236 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.408245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.408702 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.408705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.408867 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.409353 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:07.409367 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:07.410601 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:07.412027 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:07.412040 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:07.412052 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:07.412338 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:07.412612 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:07.412615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:07.412617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:07.412743 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:07.413123 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:07.413129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:07.413335 (+ 206us) client_negotiation.cc:769] Sending connection context
0809 09:30:07.413494 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 09:30:07.413761 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":170,"threads_started":1}
I20260809 09:30:07.415354 1152 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40807
I20260809 09:30:07.415575 1152 heartbeater.cc:464] Registering TS with master...
I20260809 09:30:07.416004 1152 heartbeater.cc:511] Master 127.9.102.62:40807 requested a full tablet report, sending...
I20260809 09:30:07.416913 350 ts_manager.cc:194] Registered new tserver with Master: 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373)
I20260809 09:30:07.417356 350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:57969
I20260809 09:30:07.427771 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:30:07.435935 415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.429684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.9.102.62:40807)
0809 09:30:07.429783 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:07.429786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:07.429934 (+ 148us) server_negotiation.cc:407] Connection header received
0809 09:30:07.430018 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:07.430020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:07.430051 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:07.430113 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:07.430731 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.431190 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.431940 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.432210 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.432283 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.433482 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:07.433484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:07.433486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:07.433502 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:07.434588 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:07.434941 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:07.434943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:07.434949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:07.435034 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:07.435270 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:07.435272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:07.435273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:07.435459 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:07.435566 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:07.435750 (+ 184us) server_negotiation.cc:299] Negotiation successful
0809 09:30:07.435834 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 09:30:07.438647 350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:30:07.441565 350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:30:07.454815 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.449126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:41373 (local address 127.0.0.1:49266)
0809 09:30:07.449657 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:07.449703 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:07.449809 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:07.450661 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:07.450665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:07.450683 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:07.450890 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.450900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.452353 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.452356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.453513 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.453521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.454297 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.454300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.454396 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.454434 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 09:30:07.454558 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 09:30:07.454675 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":139,"thread_start_us":68,"threads_started":1}
I20260809 09:30:07.454936 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.449237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:43639 (local address 127.0.0.1:39780)
0809 09:30:07.449932 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:07.449945 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:07.450010 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:07.450705 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:07.450707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:07.450725 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:07.450909 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.450915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.452348 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.452352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.453542 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.453551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.454569 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.454572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.454666 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.454685 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 09:30:07.454741 (+ 56us) client_negotiation.cc:240] Negotiation successful
0809 09:30:07.454789 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":44,"thread_start_us":75,"threads_started":1}
I20260809 09:30:07.455132 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.449605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.9.102.1:43639)
0809 09:30:07.450264 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:07.450276 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:07.450300 (+ 24us) server_negotiation.cc:407] Connection header received
0809 09:30:07.450395 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:07.450401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:07.450584 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:07.450728 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:07.451051 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.452198 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.453666 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.454439 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.454563 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.454711 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:07.454765 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 09:30:07.454846 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":85,"threads_started":1}
I20260809 09:30:07.455716 1177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.449215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.9.102.3:41373)
0809 09:30:07.450023 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:07.450036 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:07.450058 (+ 22us) server_negotiation.cc:407] Connection header received
0809 09:30:07.450139 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:07.450144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:07.450348 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:07.450500 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:07.451023 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.452224 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.453632 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.454173 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.454286 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.454413 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:07.455448 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0809 09:30:07.455569 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":84,"threads_started":1}
I20260809 09:30:07.456195 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.450156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.9.102.2:39087)
0809 09:30:07.450606 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:07.450617 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:07.451105 (+ 488us) server_negotiation.cc:407] Connection header received
0809 09:30:07.451173 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:07.451176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:07.451314 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:07.451469 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:07.451953 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.453293 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.454642 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.455429 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.455548 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.455665 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:07.455938 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 09:30:07.456042 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":160,"threads_started":1}
I20260809 09:30:07.456485 1182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.449753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:39087 (local address 127.0.0.1:54572)
0809 09:30:07.450889 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:07.450902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:07.450994 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:07.451749 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:07.451752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:07.451769 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:07.451841 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.451845 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.453422 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.453426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.454468 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.454481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.455552 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.455556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.455659 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.455681 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 09:30:07.456295 (+ 614us) client_negotiation.cc:240] Negotiation successful
0809 09:30:07.456370 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"thread_start_us":77,"threads_started":1}
I20260809 09:30:07.457950 573 tablet_service.cc:1467] Processing CreateTablet for tablet 55cb0237b27f4508a09d92c596e0906f (DEFAULT_TABLE table=test-table [id=00eb082e4bc14a9880ffe2ce95dd575e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:30:07.458119 1047 tablet_service.cc:1467] Processing CreateTablet for tablet 55cb0237b27f4508a09d92c596e0906f (DEFAULT_TABLE table=test-table [id=00eb082e4bc14a9880ffe2ce95dd575e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:30:07.458981 1047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cb0237b27f4508a09d92c596e0906f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:07.458979 573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cb0237b27f4508a09d92c596e0906f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:07.460652 808 tablet_service.cc:1467] Processing CreateTablet for tablet 55cb0237b27f4508a09d92c596e0906f (DEFAULT_TABLE table=test-table [id=00eb082e4bc14a9880ffe2ce95dd575e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:30:07.461534 808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55cb0237b27f4508a09d92c596e0906f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:07.465939 1190 tablet_bootstrap.cc:492] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Bootstrap starting.
I20260809 09:30:07.466969 1192 tablet_bootstrap.cc:492] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c: Bootstrap starting.
I20260809 09:30:07.467110 1194 tablet_bootstrap.cc:492] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217: Bootstrap starting.
I20260809 09:30:07.468653 1190 tablet_bootstrap.cc:654] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:07.468940 1192 tablet_bootstrap.cc:654] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:07.469338 1194 tablet_bootstrap.cc:654] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:07.469372 1190 log.cc:824] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:07.470173 1194 log.cc:824] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:07.470173 1192 log.cc:824] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:07.471516 1190 tablet_bootstrap.cc:492] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: No bootstrap required, opened a new log
I20260809 09:30:07.471705 1190 ts_tablet_manager.cc:1397] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 09:30:07.471882 1192 tablet_bootstrap.cc:492] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c: No bootstrap required, opened a new log
I20260809 09:30:07.472128 1192 ts_tablet_manager.cc:1397] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:30:07.472388 1194 tablet_bootstrap.cc:492] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217: No bootstrap required, opened a new log
I20260809 09:30:07.472549 1194 ts_tablet_manager.cc:1397] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 09:30:07.474989 1192 raft_consensus.cc:348] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } }
I20260809 09:30:07.475207 1192 raft_consensus.cc:374] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:07.475265 1192 raft_consensus.cc:729] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f72ef30dac5468493c7e47c28e0db1c, State: Initialized, Role: FOLLOWER
I20260809 09:30:07.475167 1190 raft_consensus.cc:348] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } }
I20260809 09:30:07.475437 1190 raft_consensus.cc:374] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:07.475513 1190 raft_consensus.cc:729] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ba480de48747e99a964e8f9a50f440, State: Initialized, Role: FOLLOWER
I20260809 09:30:07.475832 1192 consensus_queue.cc:260] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } }
I20260809 09:30:07.475977 1190 consensus_queue.cc:260] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } }
I20260809 09:30:07.476011 1194 raft_consensus.cc:348] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } }
I20260809 09:30:07.476313 1194 raft_consensus.cc:374] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:07.476377 1194 raft_consensus.cc:729] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0e48d0c8e246aea9a13ddd0b972217, State: Initialized, Role: FOLLOWER
I20260809 09:30:07.476644 1152 heartbeater.cc:503] Master 127.9.102.62:40807 was elected leader, sending a full tablet report...
I20260809 09:30:07.476816 1192 ts_tablet_manager.cc:1428] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:30:07.476747 1194 consensus_queue.cc:260] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } }
I20260809 09:30:07.476907 1190 ts_tablet_manager.cc:1428] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:30:07.477368 679 heartbeater.cc:503] Master 127.9.102.62:40807 was elected leader, sending a full tablet report...
I20260809 09:30:07.477612 1194 ts_tablet_manager.cc:1428] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:30:07.477931 932 heartbeater.cc:503] Master 127.9.102.62:40807 was elected leader, sending a full tablet report...
W20260809 09:30:07.505776 680 tablet.cc:2369] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:30:07.580621 1204 raft_consensus.cc:482] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:30:07.580797 1204 raft_consensus.cc:504] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } }
I20260809 09:30:07.581830 1204 leader_election.cc:288] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373), a7ba480de48747e99a964e8f9a50f440 (127.9.102.1:43639)
I20260809 09:30:07.585326 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.582146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:39411 (local address 127.9.102.1:43639)
0809 09:30:07.582286 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:07.582290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:07.582305 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:30:07.582354 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:07.582357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:07.582402 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:07.582508 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:07.582788 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.583271 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.584404 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.584778 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.584896 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.584974 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:07.585161 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 09:30:07.585218 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 09:30:07.585328 934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.582034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:43639 (local address 127.9.102.2:39411)
0809 09:30:07.582131 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:07.582145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:07.582228 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:07.582523 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:07.582526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:07.582551 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:07.582673 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.582678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.583381 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.583387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.584265 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.584275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.584899 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.584902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.585028 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.585048 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 09:30:07.585146 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 09:30:07.585213 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":29,"spinlock_wait_cycles":6400}
I20260809 09:30:07.585727 1177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.582131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:42389 (local address 127.9.102.3:41373)
0809 09:30:07.582293 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:07.582299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:07.582554 (+ 255us) server_negotiation.cc:407] Connection header received
0809 09:30:07.582666 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:07.582668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:07.582714 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:07.582798 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:07.583066 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.583532 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.584703 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.585118 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.585227 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.585281 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:07.585562 (+ 281us) server_negotiation.cc:299] Negotiation successful
0809 09:30:07.585643 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 09:30:07.585724 1205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.582033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:41373 (local address 127.9.102.2:42389)
0809 09:30:07.582442 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:07.582452 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:07.582578 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:07.582811 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:07.582813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:07.582828 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:07.582952 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.582957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.583647 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.583652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.584577 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:07.584587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.585233 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.585236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:07.585324 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.585343 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 09:30:07.585513 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 09:30:07.585589 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":114,"thread_start_us":95,"threads_started":1}
I20260809 09:30:07.586169 599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cb0237b27f4508a09d92c596e0906f" candidate_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ba480de48747e99a964e8f9a50f440" is_pre_election: true
I20260809 09:30:07.586434 1079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cb0237b27f4508a09d92c596e0906f" candidate_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f72ef30dac5468493c7e47c28e0db1c" is_pre_election: true
I20260809 09:30:07.586463 599 raft_consensus.cc:2454] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f0e48d0c8e246aea9a13ddd0b972217 in term 0.
I20260809 09:30:07.586691 1079 raft_consensus.cc:2454] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f0e48d0c8e246aea9a13ddd0b972217 in term 0.
I20260809 09:30:07.586926 726 leader_election.cc:302] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f0e48d0c8e246aea9a13ddd0b972217, a7ba480de48747e99a964e8f9a50f440; no voters:
I20260809 09:30:07.587152 1204 raft_consensus.cc:2781] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:30:07.587219 1204 raft_consensus.cc:482] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:30:07.587245 1204 raft_consensus.cc:3037] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:07.588225 1204 raft_consensus.cc:504] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } }
I20260809 09:30:07.588565 1204 leader_election.cc:288] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [CANDIDATE]: Term 1 election: Requested vote from peers 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373), a7ba480de48747e99a964e8f9a50f440 (127.9.102.1:43639)
I20260809 09:30:07.588913 1079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cb0237b27f4508a09d92c596e0906f" candidate_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f72ef30dac5468493c7e47c28e0db1c"
I20260809 09:30:07.588913 599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55cb0237b27f4508a09d92c596e0906f" candidate_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ba480de48747e99a964e8f9a50f440"
I20260809 09:30:07.589049 1079 raft_consensus.cc:3037] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:07.589057 599 raft_consensus.cc:3037] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:07.590351 599 raft_consensus.cc:2454] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f0e48d0c8e246aea9a13ddd0b972217 in term 1.
I20260809 09:30:07.590351 1079 raft_consensus.cc:2454] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f0e48d0c8e246aea9a13ddd0b972217 in term 1.
I20260809 09:30:07.590651 732 leader_election.cc:302] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f72ef30dac5468493c7e47c28e0db1c, 9f0e48d0c8e246aea9a13ddd0b972217; no voters:
I20260809 09:30:07.590811 1204 raft_consensus.cc:2781] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:30:07.591042 1204 raft_consensus.cc:686] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 1 LEADER]: Becoming Leader. State: Replica: 9f0e48d0c8e246aea9a13ddd0b972217, State: Running, Role: LEADER
I20260809 09:30:07.591375 1204 consensus_queue.cc:237] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } }
I20260809 09:30:07.594399 350 catalog_manager.cc:5495] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2). New cstate: current_term: 1 leader_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } health_report { overall_health: HEALTHY } } }
I20260809 09:30:07.603984 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.600617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.9.102.2:39087)
0809 09:30:07.600740 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:07.600743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:07.600754 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:30:07.600847 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:07.600849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:07.600912 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:07.601028 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:30:07.601334 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.602112 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.602853 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:07.603017 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:07.603081 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:07.603321 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:30:07.603717 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:30:07.603802 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:07.603864 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 09:30:07.603904 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 09:30:07.608588 599 raft_consensus.cc:1261] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Refusing update from remote peer 9f0e48d0c8e246aea9a13ddd0b972217: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:30:07.608606 1079 raft_consensus.cc:1261] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c [term 1 FOLLOWER]: Refusing update from remote peer 9f0e48d0c8e246aea9a13ddd0b972217: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:30:07.609234 1208 consensus_queue.cc:1035] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:30:07.609391 1204 consensus_queue.cc:1035] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:30:07.615433 1215 mvcc.cc:205] Tried to move back new op lower bound from 7316552939959357440 to 7316552939897466880. Current Snapshot: MvccSnapshot[applied={T|T < 7316552939959357440}]
I20260809 09:30:07.620204 1218 mvcc.cc:205] Tried to move back new op lower bound from 7316552939959357440 to 7316552939897466880. Current Snapshot: MvccSnapshot[applied={T|T < 7316552939959357440}]
I20260809 09:30:07.623358 349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:30:07.623471 349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:30:07.625618 349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Sending DeleteTablet for 3 replicas of tablet 55cb0237b27f4508a09d92c596e0906f
I20260809 09:30:07.626247 1047 tablet_service.cc:1514] Processing DeleteTablet for tablet 55cb0237b27f4508a09d92c596e0906f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:07 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:49266
I20260809 09:30:07.626391 573 tablet_service.cc:1514] Processing DeleteTablet for tablet 55cb0237b27f4508a09d92c596e0906f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:07 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:39780
I20260809 09:30:07.626711 1230 tablet_replica.cc:330] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c: stopping tablet replica
I20260809 09:30:07.626744 808 tablet_service.cc:1514] Processing DeleteTablet for tablet 55cb0237b27f4508a09d92c596e0906f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:07 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:54572
I20260809 09:30:07.626997 1230 raft_consensus.cc:2229] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:30:07.627202 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 456
I20260809 09:30:07.627292 1235 tablet_replica.cc:330] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217: stopping tablet replica
I20260809 09:30:07.627609 1235 raft_consensus.cc:2229] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:30:07.628189 1230 raft_consensus.cc:2258] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:30:07.633751 1230 ts_tablet_manager.cc:1905] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:30:07.634958 1235 raft_consensus.cc:2258] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:30:07.636579 1230 ts_tablet_manager.cc:1918] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:30:07.636672 1230 log.cc:1196] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/55cb0237b27f4508a09d92c596e0906f
I20260809 09:30:07.636956 1230 ts_tablet_manager.cc:1939] T 55cb0237b27f4508a09d92c596e0906f P 4f72ef30dac5468493c7e47c28e0db1c: Deleting consensus metadata
I20260809 09:30:07.638024 328 catalog_manager.cc:4841] TS 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373): tablet 55cb0237b27f4508a09d92c596e0906f (table test-table [id=00eb082e4bc14a9880ffe2ce95dd575e]) successfully deleted
I20260809 09:30:07.638417 1235 ts_tablet_manager.cc:1905] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:30:07.640417 1235 ts_tablet_manager.cc:1918] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:30:07.640543 1235 log.cc:1196] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/55cb0237b27f4508a09d92c596e0906f
I20260809 09:30:07.640826 1235 ts_tablet_manager.cc:1939] T 55cb0237b27f4508a09d92c596e0906f P 9f0e48d0c8e246aea9a13ddd0b972217: Deleting consensus metadata
I20260809 09:30:07.641786 325 catalog_manager.cc:4841] TS 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087): tablet 55cb0237b27f4508a09d92c596e0906f (table test-table [id=00eb082e4bc14a9880ffe2ce95dd575e]) successfully deleted
W20260809 09:30:07.649380 325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:43639 (error 108)
I20260809 09:30:07.649396 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 692
I20260809 09:30:07.654256 1182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:07.653852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:43639 (local address 127.0.0.1:39790)
0809 09:30:07.654114 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:07.654171 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
W20260809 09:30:07.654397 325 catalog_manager.cc:4584] TS a7ba480de48747e99a964e8f9a50f440 (127.9.102.1:43639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55cb0237b27f4508a09d92c596e0906f: Network error: Client connection negotiation failed: client connection to 127.9.102.1:43639: connect: Connection refused (error 111)
I20260809 09:30:07.661336 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 940
I20260809 09:30:07.668135 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 32763
I20260809 09:30:07.677881 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40807
--webserver_interface=127.9.102.62
--webserver_port=35297
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:40807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:07.776062 1237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:07.776268 1237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:07.776301 1237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:07.779911 1237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:30:07.779956 1237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:07.779973 1237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:07.779987 1237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:30:07.780000 1237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:30:07.784632 1237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:40807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=35297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:07.785383 1237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:07.786229 1237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:07.798758 1248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:07.798754 1251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:07.798754 1249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:07.798952 1237 server_base.cc:1034] running on GCE node
I20260809 09:30:07.799608 1237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:07.800716 1237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:07.801877 1237 hybrid_clock.cc:648] HybridClock initialized: now 1786267807801863 us; error 24 us; skew 500 ppm
Aug 09 09:30:07 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267807, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:07.804508 1237 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:30:07.805367 1237 webserver.cc:460] Webserver started at http://127.9.102.62:35297/ using document root <none> and password file <none>
I20260809 09:30:07.805905 1237 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:07.806018 1237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:07.809492 1237 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:30:07.811447 1267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:07.812606 1237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:07.812740 1237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "d77cb80bc89c4cd2baea34c10760dc76"
format_stamp: "Formatted at 2026-08-09 09:30:06 on dist-test-slave-7vjv"
server_key: "1de7dfecd8991203b253d0de0b3b2755"
server_key_iv: "3f46d4e2f1eee7d06658656552b924dc"
server_key_version: "encryptionkey@0"
I20260809 09:30:07.813270 1237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:07.825770 1237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:07.828145 1237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:07.828305 1237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:07.834703 1237 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:40807
I20260809 09:30:07.835049 1369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:40807 every 8 connection(s)
I20260809 09:30:07.835729 1237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:30:07.839555 1372 sys_catalog.cc:263] Verifying existing consensus state
I20260809 09:30:07.841202 1372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Bootstrap starting.
I20260809 09:30:07.844246 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 1237
I20260809 09:30:07.844708 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:43639
--local_ip_for_outbound_sockets=127.9.102.1
--tserver_master_addrs=127.9.102.62:40807
--webserver_port=38097
--webserver_interface=127.9.102.1
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:30:07.858191 1372 log.cc:824] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:07.866058 1372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 09:30:07.866387 1372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Bootstrap complete.
I20260809 09:30:07.870169 1372 raft_consensus.cc:348] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } }
I20260809 09:30:07.870523 1372 raft_consensus.cc:729] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77cb80bc89c4cd2baea34c10760dc76, State: Initialized, Role: FOLLOWER
I20260809 09:30:07.871033 1372 consensus_queue.cc:260] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } }
I20260809 09:30:07.871152 1372 raft_consensus.cc:388] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:30:07.871238 1372 raft_consensus.cc:482] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:30:07.871352 1372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 1 FOLLOWER]: Advancing to term 2
I20260809 09:30:07.873059 1372 raft_consensus.cc:504] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } }
I20260809 09:30:07.873361 1372 leader_election.cc:302] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d77cb80bc89c4cd2baea34c10760dc76; no voters:
I20260809 09:30:07.873627 1372 leader_election.cc:288] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 09:30:07.873739 1382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 09:30:07.873955 1382 raft_consensus.cc:686] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 2 LEADER]: Becoming Leader. State: Replica: d77cb80bc89c4cd2baea34c10760dc76, State: Running, Role: LEADER
I20260809 09:30:07.874226 1382 consensus_queue.cc:237] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } }
I20260809 09:30:07.874704 1372 sys_catalog.cc:564] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:30:07.875419 1384 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d77cb80bc89c4cd2baea34c10760dc76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } } }
I20260809 09:30:07.875553 1384 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:07.875865 1385 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d77cb80bc89c4cd2baea34c10760dc76. Latest consensus state: current_term: 2 leader_uuid: "d77cb80bc89c4cd2baea34c10760dc76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } } }
I20260809 09:30:07.875944 1385 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:07.876293 1400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:30:07.880487 1400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:30:07.881047 1400 catalog_manager.cc:1259] Loaded cluster ID: d67cfc51a1c64a27872e89584dfb15de
I20260809 09:30:07.881100 1400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:30:07.882334 1400 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 09:30:07.886725 1400 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 09:30:07.891484 1400 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:30:07.892179 1400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Loaded TSK: 0
I20260809 09:30:07.893313 1400 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 09:30:07.952085 1380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:07.952365 1380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:07.952401 1380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:07.955940 1380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:07.955986 1380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:07.956059 1380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:30:07.960628 1380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:43639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=38097
--tserver_master_addrs=127.9.102.62:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:07.961447 1380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:07.962272 1380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:07.970772 1429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:07.970811 1431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:07.970777 1439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:07.971275 1380 server_base.cc:1034] running on GCE node
I20260809 09:30:07.971629 1380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:07.972242 1380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:07.973431 1380 hybrid_clock.cc:648] HybridClock initialized: now 1786267807973404 us; error 45 us; skew 500 ppm
Aug 09 09:30:07 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267807, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:07.976406 1380 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:30:07.977277 1380 webserver.cc:460] Webserver started at http://127.9.102.1:38097/ using document root <none> and password file <none>
I20260809 09:30:07.977820 1380 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:07.977933 1380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:07.981624 1380 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:30:07.983781 1450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:07.985037 1380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:07.985164 1380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "a7ba480de48747e99a964e8f9a50f440"
format_stamp: "Formatted at 2026-08-09 09:30:06 on dist-test-slave-7vjv"
server_key: "61cb33e56884fc6d8b350d2b008fccdb"
server_key_iv: "c6e541cb9e5b884c94c7cc4bb2390427"
server_key_version: "encryptionkey@0"
I20260809 09:30:07.985636 1380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:08.004796 1380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:08.007741 1380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:08.007925 1380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:08.008492 1380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:08.009496 1462 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 09:30:08.013204 1380 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 09:30:08.013278 1380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260809 09:30:08.013358 1380 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 09:30:08.014856 1380 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 09:30:08.014922 1380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260809 09:30:08.015190 1462 tablet_bootstrap.cc:492] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Bootstrap starting.
I20260809 09:30:08.024608 1380 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:43639
I20260809 09:30:08.024706 1652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:43639 every 8 connection(s)
I20260809 09:30:08.025439 1380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:30:08.031405 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 1380
I20260809 09:30:08.031833 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:39087
--local_ip_for_outbound_sockets=127.9.102.2
--tserver_master_addrs=127.9.102.62:40807
--webserver_port=34289
--webserver_interface=127.9.102.2
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:30:08.032935 1462 log.cc:824] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Log is configured to *not* fsync() on all Append() calls
Aug 09 09:30:08 dist-test-slave-7vjv krb5kdc[32739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267807, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:08.040079 1462 tablet_bootstrap.cc:492] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 09:30:08.040542 1462 tablet_bootstrap.cc:492] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Bootstrap complete.
I20260809 09:30:08.041769 1462 ts_tablet_manager.cc:1397] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.004s
I20260809 09:30:08.044014 1462 raft_consensus.cc:348] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } }
I20260809 09:30:08.044553 1462 raft_consensus.cc:729] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ba480de48747e99a964e8f9a50f440, State: Initialized, Role: FOLLOWER
I20260809 09:30:08.045073 1462 consensus_queue.cc:260] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } }
I20260809 09:30:08.045701 1656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.027287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:39671 (local address 127.9.102.62:40807)
0809 09:30:08.027625 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.027639 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.028088 (+ 449us) server_negotiation.cc:407] Connection header received
0809 09:30:08.029158 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.029168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.029433 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.029782 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:08.030819 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.032057 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.033730 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.034460 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.034643 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.037935 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:08.037944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:08.037962 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:08.038000 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:08.040358 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:08.041020 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:08.041038 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:08.041045 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:08.042765 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:08.043170 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:08.043175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:08.043178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:08.043666 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:08.043816 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.044481 (+ 665us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.044818 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":119,"threads_started":1}
I20260809 09:30:08.045712 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.027376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:39671)
0809 09:30:08.027902 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:08.027961 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:08.028775 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:08.029830 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:08.029837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:08.030222 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:08.030652 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.030664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.032237 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.032241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.033558 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.033572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.034866 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.034872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.035106 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.035809 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:08.035835 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:08.037723 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:08.040501 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:08.040506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:08.040521 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:08.040890 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:08.042886 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:08.042889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:08.042890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:08.043056 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:08.043800 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:08.043805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:08.044097 (+ 292us) client_negotiation.cc:769] Sending connection context
0809 09:30:08.044323 (+ 226us) client_negotiation.cc:240] Negotiation successful
0809 09:30:08.044755 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":174,"threads_started":1}
I20260809 09:30:08.045980 1462 ts_tablet_manager.cc:1428] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260809 09:30:08.048153 1654 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40807
I20260809 09:30:08.048373 1654 heartbeater.cc:464] Registering TS with master...
I20260809 09:30:08.048776 1654 heartbeater.cc:511] Master 127.9.102.62:40807 requested a full tablet report, sending...
I20260809 09:30:08.050843 1292 ts_manager.cc:194] Registered new tserver with Master: a7ba480de48747e99a964e8f9a50f440 (127.9.102.1:43639)
I20260809 09:30:08.053876 1292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:39671
I20260809 09:30:08.056196 1654 heartbeater.cc:503] Master 127.9.102.62:40807 was elected leader, sending a full tablet report...
Aug 09 09:30:08 dist-test-slave-7vjv krb5kdc[32739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267807, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.1@KRBTEST.COM
I20260809 09:30:08.062994 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.053388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.9.102.1:43639)
0809 09:30:08.053705 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.053715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.054068 (+ 353us) server_negotiation.cc:407] Connection header received
0809 09:30:08.054257 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.054262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.054422 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.054553 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:08.055172 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.056188 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.057349 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.057696 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.057770 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.059904 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:08.059912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:08.059920 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:08.059938 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:08.061296 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:08.061804 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:08.061808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:08.061809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:08.061877 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:08.062139 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:08.062141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:08.062142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:08.062361 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:08.062446 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.062797 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.062903 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":86,"threads_started":1}
I20260809 09:30:08.063020 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.053292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:43639 (local address 127.0.0.1:39798)
0809 09:30:08.053949 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:08.053992 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:08.054110 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:08.054578 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:08.054583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:08.054932 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:08.055048 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.055056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.056320 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.056324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.057208 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.057219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.057815 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.057819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.057916 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.058350 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:08.058363 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:08.059776 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:08.061403 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:08.061411 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:08.061418 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:08.061698 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:08.061960 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:08.061963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:08.061965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:08.062056 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:08.062457 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:08.062462 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:08.062596 (+ 134us) client_negotiation.cc:769] Sending connection context
0809 09:30:08.062736 (+ 140us) client_negotiation.cc:240] Negotiation successful
0809 09:30:08.062888 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":92,"threads_started":1}
I20260809 09:30:08.064076 1532 tablet_service.cc:1514] Processing DeleteTablet for tablet 55cb0237b27f4508a09d92c596e0906f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:07 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:39798
I20260809 09:30:08.064836 1672 tablet_replica.cc:330] stopping tablet replica
I20260809 09:30:08.065119 1672 raft_consensus.cc:2229] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:30:08.065356 1672 raft_consensus.cc:2258] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:30:08.066107 1672 ts_tablet_manager.cc:1905] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:30:08.069190 1672 ts_tablet_manager.cc:1918] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:30:08.069260 1672 log.cc:1196] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal/wals/55cb0237b27f4508a09d92c596e0906f
I20260809 09:30:08.069518 1672 ts_tablet_manager.cc:1939] T 55cb0237b27f4508a09d92c596e0906f P a7ba480de48747e99a964e8f9a50f440: Deleting consensus metadata
I20260809 09:30:08.070885 1269 catalog_manager.cc:4841] TS a7ba480de48747e99a964e8f9a50f440 (127.9.102.1:43639): tablet 55cb0237b27f4508a09d92c596e0906f (table test-table [id=00eb082e4bc14a9880ffe2ce95dd575e]) successfully deleted
W20260809 09:30:08.139235 1662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:08.139463 1662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:08.139539 1662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:08.143260 1662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:08.143338 1662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:08.143414 1662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:30:08.147867 1662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:39087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=34289
--tserver_master_addrs=127.9.102.62:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:08.148715 1662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:08.149564 1662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:08.157656 1682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:08.157644 1685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:08.157657 1681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:08.157763 1662 server_base.cc:1034] running on GCE node
I20260809 09:30:08.158336 1662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:08.158981 1662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:08.160167 1662 hybrid_clock.cc:648] HybridClock initialized: now 1786267808160139 us; error 28 us; skew 500 ppm
Aug 09 09:30:08 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267808, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:08.162604 1662 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:30:08.163375 1662 webserver.cc:460] Webserver started at http://127.9.102.2:34289/ using document root <none> and password file <none>
I20260809 09:30:08.163852 1662 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:08.163954 1662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:08.167191 1662 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260809 09:30:08.168962 1693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:08.169998 1662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:08.170089 1662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "9f0e48d0c8e246aea9a13ddd0b972217"
format_stamp: "Formatted at 2026-08-09 09:30:07 on dist-test-slave-7vjv"
server_key: "5dddd06c6c393a9c1a1d2ca956cd2be1"
server_key_iv: "79b8d7ce1a6d8f80133c42f316f69d01"
server_key_version: "encryptionkey@0"
I20260809 09:30:08.170481 1662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:08.184580 1662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:08.187244 1662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:08.187364 1662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:08.187758 1662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:08.188519 1662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:30:08.188560 1662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:08.188591 1662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:30:08.188612 1662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:08.196910 1662 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:39087
I20260809 09:30:08.197213 1883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:39087 every 8 connection(s)
I20260809 09:30:08.197827 1662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:30:08.199256 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 1662
I20260809 09:30:08.199692 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:41373
--local_ip_for_outbound_sockets=127.9.102.3
--tserver_master_addrs=127.9.102.62:40807
--webserver_port=38765
--webserver_interface=127.9.102.3
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 09:30:08 dist-test-slave-7vjv krb5kdc[32739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267808, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:08.210047 1656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.199375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:58269 (local address 127.9.102.62:40807)
0809 09:30:08.199508 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.199514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.200263 (+ 749us) server_negotiation.cc:407] Connection header received
0809 09:30:08.201027 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.201033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.201096 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.201187 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:08.202524 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.202978 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.204039 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.204359 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.204466 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.206950 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:08.206955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:08.206962 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:08.206983 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:08.208290 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:08.208734 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:08.208737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:08.208739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:08.208811 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:08.209113 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:08.209116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:08.209118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:08.209272 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:08.209368 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.209774 (+ 406us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.209898 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 09:30:08.210650 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.199673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:58269)
0809 09:30:08.200120 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:08.200166 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:08.200842 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:08.201334 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:08.201339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:08.201812 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:08.202376 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.202394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.203099 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.203102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.203877 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.203887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.204541 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.204547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.204763 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.205274 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:08.205299 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:08.206804 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:08.208395 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:08.208400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:08.208412 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:08.208638 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:08.208895 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:08.208898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:08.208900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:08.209021 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:08.209376 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:08.209383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:08.209577 (+ 194us) client_negotiation.cc:769] Sending connection context
0809 09:30:08.209735 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 09:30:08.209993 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":136,"threads_started":1}
I20260809 09:30:08.211704 1884 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40807
I20260809 09:30:08.211898 1884 heartbeater.cc:464] Registering TS with master...
I20260809 09:30:08.212306 1884 heartbeater.cc:511] Master 127.9.102.62:40807 requested a full tablet report, sending...
I20260809 09:30:08.213258 1292 ts_manager.cc:194] Registered new tserver with Master: 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087)
I20260809 09:30:08.213793 1292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:58269
W20260809 09:30:08.300498 1889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:08.300770 1889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:08.300844 1889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:08.304322 1889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:08.304394 1889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:08.304474 1889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:30:08.309113 1889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:41373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=38765
--tserver_master_addrs=127.9.102.62:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:08.309983 1889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:08.310868 1889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:08.319819 1899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:08.319862 1889 server_base.cc:1034] running on GCE node
W20260809 09:30:08.319806 1907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:08.319785 1901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:08.320446 1889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:08.320978 1889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:08.322157 1889 hybrid_clock.cc:648] HybridClock initialized: now 1786267808322139 us; error 28 us; skew 500 ppm
Aug 09 09:30:08 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267808, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:08.324666 1889 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:30:08.325399 1889 webserver.cc:460] Webserver started at http://127.9.102.3:38765/ using document root <none> and password file <none>
I20260809 09:30:08.325910 1889 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:08.326007 1889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:08.329412 1889 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:30:08.331282 1924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:08.332548 1889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:08.332664 1889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "4f72ef30dac5468493c7e47c28e0db1c"
format_stamp: "Formatted at 2026-08-09 09:30:07 on dist-test-slave-7vjv"
server_key: "bb5277e2a448d063d3cd0074b6c18b71"
server_key_iv: "c9003a26966708592c358093f97183f7"
server_key_version: "encryptionkey@0"
I20260809 09:30:08.333112 1889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:08.347219 1889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:08.350159 1889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:08.350356 1889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:08.350879 1889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:08.351675 1889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:30:08.351812 1889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:08.351965 1889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:30:08.352023 1889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:08.361176 1889 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:41373
I20260809 09:30:08.361194 2117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:41373 every 8 connection(s)
I20260809 09:30:08.362072 1889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:30:08.366864 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 1889
Aug 09 09:30:08 dist-test-slave-7vjv krb5kdc[32739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267808, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:08.376191 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.367502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.9.102.62:40807)
0809 09:30:08.367741 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.367743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.368011 (+ 268us) server_negotiation.cc:407] Connection header received
0809 09:30:08.368156 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.368158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.368198 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.368268 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:08.369042 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.369711 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.370789 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.371060 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.371173 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.372544 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:08.372548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:08.372551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:08.372594 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:08.374255 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:08.374708 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:08.374711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:08.374713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:08.374792 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:08.375135 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:08.375137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:08.375139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:08.375302 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:08.375496 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.375823 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.375953 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":91,"threads_started":1}
I20260809 09:30:08.376508 1656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.363638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:49825 (local address 127.9.102.62:40807)
0809 09:30:08.363763 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.363766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.364566 (+ 800us) server_negotiation.cc:407] Connection header received
0809 09:30:08.365785 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.365787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.365856 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.365936 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:08.367115 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.367789 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.369238 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.369566 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.369665 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.372261 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:08.372271 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:08.372274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:08.372297 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:08.374012 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:08.374527 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:08.374529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:08.374530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:08.374703 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:08.375078 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:08.375082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:08.375087 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:08.375275 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:08.375356 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.375960 (+ 604us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.376381 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 09:30:08.377250 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.363981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:49825)
0809 09:30:08.364433 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:08.364467 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:08.365567 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:08.366088 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:08.366093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:08.366395 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:08.366953 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.366974 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.367913 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.367917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.369137 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.369146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.369703 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.369705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.369873 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.370366 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:08.370383 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:08.372049 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:08.374123 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:08.374127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:08.374137 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:08.374421 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:08.374835 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:08.374839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:08.374841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:08.374965 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:08.375383 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:08.375390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:08.375717 (+ 327us) client_negotiation.cc:769] Sending connection context
0809 09:30:08.375904 (+ 187us) client_negotiation.cc:240] Negotiation successful
0809 09:30:08.376205 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":167,"threads_started":1}
I20260809 09:30:08.378434 2118 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40807
I20260809 09:30:08.378602 2118 heartbeater.cc:464] Registering TS with master...
I20260809 09:30:08.378954 2118 heartbeater.cc:511] Master 127.9.102.62:40807 requested a full tablet report, sending...
I20260809 09:30:08.379791 1292 ts_manager.cc:194] Registered new tserver with Master: 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373)
I20260809 09:30:08.380393 1292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:49825
I20260809 09:30:08.389569 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:30:08.393117 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.390065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.9.102.62:40807)
0809 09:30:08.390167 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.390171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.390326 (+ 155us) server_negotiation.cc:407] Connection header received
0809 09:30:08.390414 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.390416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.390489 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.390562 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:30:08.390746 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.391157 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.391902 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.392171 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.392259 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.392498 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:30:08.392783 (+ 285us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:30:08.392861 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.392919 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.392961 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 09:30:08.393633 1292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:30:08.396749 1292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:30:08.404013 1532 tablet_service.cc:1467] Processing CreateTablet for tablet 73d48ab7f63f488bb89c5f7890919a5d (DEFAULT_TABLE table=test-table [id=6e7560c6d30b4397a15e73c0dcc241fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:30:08.404719 1532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d48ab7f63f488bb89c5f7890919a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:08.407359 1462 tablet_bootstrap.cc:492] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440: Bootstrap starting.
I20260809 09:30:08.407756 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.402815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:41373 (local address 127.0.0.1:49276)
0809 09:30:08.402915 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:08.402928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:08.403012 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:08.403889 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:08.403892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:08.403909 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:08.403988 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.403993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.405710 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.405716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.406493 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.406502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.407377 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.407382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.407472 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.407529 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 09:30:08.407579 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 09:30:08.407647 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":19}
I20260809 09:30:08.408265 1462 tablet_bootstrap.cc:654] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:08.408484 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.402929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.9.102.3:41373)
0809 09:30:08.403451 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.403462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.403483 (+ 21us) server_negotiation.cc:407] Connection header received
0809 09:30:08.403553 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.403557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.403732 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.403875 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:08.404125 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.405577 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.406616 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.407252 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.407999 (+ 747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.408179 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.408230 (+ 51us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.408342 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":190,"threads_started":1}
I20260809 09:30:08.408515 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.403163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:39087 (local address 127.0.0.1:54594)
0809 09:30:08.403565 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:08.403577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:08.403652 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:08.404416 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:08.404419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:08.404438 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:08.404524 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.404530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.405753 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.405756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.406783 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.406793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.408030 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.408034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.408170 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.408211 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 09:30:08.408303 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 09:30:08.408377 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":55,"threads_started":1}
I20260809 09:30:08.408586 2137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.403382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.9.102.2:39087)
0809 09:30:08.403896 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.403907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.403928 (+ 21us) server_negotiation.cc:407] Connection header received
0809 09:30:08.403996 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.404000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.404251 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.404438 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:08.404629 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.405627 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.406902 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.407865 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.408010 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.408179 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.408341 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.408427 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":100,"threads_started":1}
I20260809 09:30:08.409857 1462 tablet_bootstrap.cc:492] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440: No bootstrap required, opened a new log
I20260809 09:30:08.409943 1462 ts_tablet_manager.cc:1397] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260809 09:30:08.410327 1462 raft_consensus.cc:348] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:08.410460 1462 raft_consensus.cc:374] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:08.410499 1462 raft_consensus.cc:729] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ba480de48747e99a964e8f9a50f440, State: Initialized, Role: FOLLOWER
I20260809 09:30:08.410655 1462 consensus_queue.cc:260] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:08.410981 1462 ts_tablet_manager.cc:1428] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260809 09:30:08.411558 1779 tablet_service.cc:1467] Processing CreateTablet for tablet 73d48ab7f63f488bb89c5f7890919a5d (DEFAULT_TABLE table=test-table [id=6e7560c6d30b4397a15e73c0dcc241fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:30:08.411561 2000 tablet_service.cc:1467] Processing CreateTablet for tablet 73d48ab7f63f488bb89c5f7890919a5d (DEFAULT_TABLE table=test-table [id=6e7560c6d30b4397a15e73c0dcc241fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:30:08.412492 1779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d48ab7f63f488bb89c5f7890919a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:08.412566 2000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d48ab7f63f488bb89c5f7890919a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:08.418694 2139 tablet_bootstrap.cc:492] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c: Bootstrap starting.
I20260809 09:30:08.420439 2144 tablet_bootstrap.cc:492] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217: Bootstrap starting.
I20260809 09:30:08.420957 2139 tablet_bootstrap.cc:654] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:08.421609 2139 log.cc:824] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:08.422590 2144 tablet_bootstrap.cc:654] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:08.423208 2139 tablet_bootstrap.cc:492] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c: No bootstrap required, opened a new log
I20260809 09:30:08.423276 2144 log.cc:824] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:08.423381 2139 ts_tablet_manager.cc:1397] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 09:30:08.425199 2144 tablet_bootstrap.cc:492] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217: No bootstrap required, opened a new log
I20260809 09:30:08.425369 2144 ts_tablet_manager.cc:1397] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 09:30:08.426450 2139 raft_consensus.cc:348] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:08.426651 2139 raft_consensus.cc:374] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:08.426714 2139 raft_consensus.cc:729] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f72ef30dac5468493c7e47c28e0db1c, State: Initialized, Role: FOLLOWER
I20260809 09:30:08.427059 2139 consensus_queue.cc:260] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:08.427625 2118 heartbeater.cc:503] Master 127.9.102.62:40807 was elected leader, sending a full tablet report...
I20260809 09:30:08.427938 2139 ts_tablet_manager.cc:1428] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260809 09:30:08.428110 2144 raft_consensus.cc:348] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:08.428340 2144 raft_consensus.cc:374] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:08.428396 2144 raft_consensus.cc:729] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0e48d0c8e246aea9a13ddd0b972217, State: Initialized, Role: FOLLOWER
I20260809 09:30:08.428781 2144 consensus_queue.cc:260] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:08.429347 1884 heartbeater.cc:503] Master 127.9.102.62:40807 was elected leader, sending a full tablet report...
I20260809 09:30:08.429656 2144 ts_tablet_manager.cc:1428] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260809 09:30:08.448655 1885 tablet.cc:2369] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:30:08.526654 1655 tablet.cc:2369] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:30:08.598522 1666 raft_consensus.cc:482] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:30:08.598704 1666 raft_consensus.cc:504] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:08.599690 1666 leader_election.cc:288] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373), 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087)
I20260809 09:30:08.602542 2151 raft_consensus.cc:482] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:30:08.602699 2151 raft_consensus.cc:504] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:08.603403 2137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.600040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:48147 (local address 127.9.102.2:39087)
0809 09:30:08.600186 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.600190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.600333 (+ 143us) server_negotiation.cc:407] Connection header received
0809 09:30:08.600473 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.600476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.600520 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.600599 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:08.600816 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.601353 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.602273 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.602856 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.602958 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.603015 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.603228 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.603297 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 09:30:08.603430 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.599952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:60725 (local address 127.9.102.3:41373)
0809 09:30:08.600108 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.600113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.600225 (+ 112us) server_negotiation.cc:407] Connection header received
0809 09:30:08.600407 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.600409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.600452 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.600523 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:08.600777 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.601267 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.602256 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.602789 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.602907 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.602974 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.603269 (+ 295us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.603335 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 09:30:08.603420 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.599887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:39087 (local address 127.9.102.1:48147)
0809 09:30:08.600252 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:08.600263 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:08.600371 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:08.600610 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:08.600612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:08.600628 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:08.600720 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.600724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.601452 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.601455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.602158 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.602166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.602961 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.602965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.603054 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.603118 (+ 64us) client_negotiation.cc:769] Sending connection context
0809 09:30:08.603201 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 09:30:08.603280 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260809 09:30:08.603828 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.599843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:41373 (local address 127.9.102.1:60725)
0809 09:30:08.600128 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:08.600143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:08.600279 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:08.600556 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:08.600559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:08.600575 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:08.600675 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.600679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.601371 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.601375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.602131 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.602140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.603021 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.603033 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.603131 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.603157 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 09:30:08.603609 (+ 452us) client_negotiation.cc:240] Negotiation successful
0809 09:30:08.603681 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":24,"thread_start_us":64,"threads_started":1}
I20260809 09:30:08.603834 2151 leader_election.cc:288] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373), a7ba480de48747e99a964e8f9a50f440 (127.9.102.1:43639)
I20260809 09:30:08.604280 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d48ab7f63f488bb89c5f7890919a5d" candidate_uuid: "a7ba480de48747e99a964e8f9a50f440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" is_pre_election: true
I20260809 09:30:08.604388 2034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d48ab7f63f488bb89c5f7890919a5d" candidate_uuid: "a7ba480de48747e99a964e8f9a50f440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f72ef30dac5468493c7e47c28e0db1c" is_pre_election: true
I20260809 09:30:08.604475 1814 raft_consensus.cc:2454] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ba480de48747e99a964e8f9a50f440 in term 0.
I20260809 09:30:08.604645 2034 raft_consensus.cc:2454] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ba480de48747e99a964e8f9a50f440 in term 0.
I20260809 09:30:08.605068 1453 leader_election.cc:302] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f72ef30dac5468493c7e47c28e0db1c, a7ba480de48747e99a964e8f9a50f440; no voters:
I20260809 09:30:08.605334 1666 raft_consensus.cc:2781] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:30:08.605418 1666 raft_consensus.cc:482] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:30:08.605470 1666 raft_consensus.cc:3037] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:08.606730 1666 raft_consensus.cc:504] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:08.607028 1666 leader_election.cc:288] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 1 election: Requested vote from peers 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373), 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087)
I20260809 09:30:08.607340 2034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d48ab7f63f488bb89c5f7890919a5d" candidate_uuid: "a7ba480de48747e99a964e8f9a50f440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f72ef30dac5468493c7e47c28e0db1c"
I20260809 09:30:08.607451 2034 raft_consensus.cc:3037] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:08.607626 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d48ab7f63f488bb89c5f7890919a5d" candidate_uuid: "a7ba480de48747e99a964e8f9a50f440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0e48d0c8e246aea9a13ddd0b972217"
I20260809 09:30:08.607726 1814 raft_consensus.cc:3037] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:08.608605 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.604138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:43639 (local address 127.9.102.2:47575)
0809 09:30:08.604238 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:08.604251 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:08.604321 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:08.605142 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:08.605147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:08.605167 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:08.605265 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.605270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.606284 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.606287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.607372 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.607382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.608218 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.608221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.608315 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.608365 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 09:30:08.608417 (+ 52us) client_negotiation.cc:240] Negotiation successful
0809 09:30:08.608484 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":3}
I20260809 09:30:08.608728 2034 raft_consensus.cc:2454] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ba480de48747e99a964e8f9a50f440 in term 1.
I20260809 09:30:08.609004 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.604717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:49369 (local address 127.9.102.3:41373)
0809 09:30:08.604829 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.604833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.604974 (+ 141us) server_negotiation.cc:407] Connection header received
0809 09:30:08.605154 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.605158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.605195 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.605235 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:08.605662 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.606361 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.607573 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.608075 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.608180 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.608229 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.608621 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.608871 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:30:08.609014 1814 raft_consensus.cc:2454] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ba480de48747e99a964e8f9a50f440 in term 1.
I20260809 09:30:08.609115 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.604338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:47575 (local address 127.9.102.1:43639)
0809 09:30:08.604939 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.604943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.604957 (+ 14us) server_negotiation.cc:407] Connection header received
0809 09:30:08.604990 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.604992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.605027 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.605363 (+ 336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:08.605438 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.606154 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.607522 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.608064 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.608781 (+ 717us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.608911 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.608948 (+ 37us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.608992 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":69,"threads_started":1}
I20260809 09:30:08.609341 1451 leader_election.cc:302] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f0e48d0c8e246aea9a13ddd0b972217, a7ba480de48747e99a964e8f9a50f440; no voters:
I20260809 09:30:08.609549 1666 raft_consensus.cc:2781] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:30:08.609571 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.604634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:41373 (local address 127.9.102.2:49369)
0809 09:30:08.604886 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:08.604897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:08.605045 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:08.605355 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:08.605357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:08.605375 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:08.605501 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.605508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.606456 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.606460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.607439 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:08.607447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.608313 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.608316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:08.608479 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.608498 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 09:30:08.609369 (+ 871us) client_negotiation.cc:240] Negotiation successful
0809 09:30:08.609436 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":47,"thread_start_us":98,"threads_started":1}
I20260809 09:30:08.609524 1566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d48ab7f63f488bb89c5f7890919a5d" candidate_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ba480de48747e99a964e8f9a50f440" is_pre_election: true
I20260809 09:30:08.609683 1666 raft_consensus.cc:686] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 LEADER]: Becoming Leader. State: Replica: a7ba480de48747e99a964e8f9a50f440, State: Running, Role: LEADER
I20260809 09:30:08.609894 2034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d48ab7f63f488bb89c5f7890919a5d" candidate_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f72ef30dac5468493c7e47c28e0db1c" is_pre_election: true
I20260809 09:30:08.609884 1666 consensus_queue.cc:237] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:08.610030 2034 raft_consensus.cc:2379] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f0e48d0c8e246aea9a13ddd0b972217 in current term 1: Already voted for candidate a7ba480de48747e99a964e8f9a50f440 in this term.
I20260809 09:30:08.612191 1694 leader_election.cc:302] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f0e48d0c8e246aea9a13ddd0b972217; no voters: 4f72ef30dac5468493c7e47c28e0db1c, a7ba480de48747e99a964e8f9a50f440
I20260809 09:30:08.612418 2151 raft_consensus.cc:2726] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 09:30:08.613119 2119 tablet.cc:2369] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:30:08.612934 1289 catalog_manager.cc:5495] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7ba480de48747e99a964e8f9a50f440 (127.9.102.1). New cstate: current_term: 1 leader_uuid: "a7ba480de48747e99a964e8f9a50f440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } health_report { overall_health: HEALTHY } } }
I20260809 09:30:08.645285 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.642181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.9.102.1:43639)
0809 09:30:08.642302 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:08.642305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:08.642317 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:30:08.642453 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:08.642456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:08.642508 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:08.642616 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:30:08.642840 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.643262 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.643992 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:08.644216 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:08.644318 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:08.644569 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:30:08.644904 (+ 335us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:30:08.645056 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:08.645127 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 09:30:08.645169 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 09:30:08.648422 2034 raft_consensus.cc:1261] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c [term 1 FOLLOWER]: Refusing update from remote peer a7ba480de48747e99a964e8f9a50f440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:30:08.648463 1814 raft_consensus.cc:1261] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [term 1 FOLLOWER]: Refusing update from remote peer a7ba480de48747e99a964e8f9a50f440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:30:08.649034 2159 consensus_queue.cc:1035] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:30:08.649178 1666 consensus_queue.cc:1035] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:30:08.656236 2165 mvcc.cc:205] Tried to move back new op lower bound from 7316552944218865664 to 7316552944068378624. Current Snapshot: MvccSnapshot[applied={T|T < 7316552944218865664}]
I20260809 09:30:08.662086 1289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:30:08.662189 1289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:30:08.664161 1289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Sending DeleteTablet for 3 replicas of tablet 73d48ab7f63f488bb89c5f7890919a5d
I20260809 09:30:08.664755 1532 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d48ab7f63f488bb89c5f7890919a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:08 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:39798
I20260809 09:30:08.664749 2000 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d48ab7f63f488bb89c5f7890919a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:08 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:49276
I20260809 09:30:08.664850 1779 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d48ab7f63f488bb89c5f7890919a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:08 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:54594
I20260809 09:30:08.665246 2187 tablet_replica.cc:330] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c: stopping tablet replica
I20260809 09:30:08.665349 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 1380
I20260809 09:30:08.665542 2187 raft_consensus.cc:2229] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:30:08.668099 2187 raft_consensus.cc:2258] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:30:08.665390 2189 tablet_replica.cc:330] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217: stopping tablet replica
I20260809 09:30:08.668540 2189 raft_consensus.cc:2229] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:30:08.668877 2189 raft_consensus.cc:2258] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:30:08.671278 2187 ts_tablet_manager.cc:1905] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:30:08.673177 2189 ts_tablet_manager.cc:1905] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:30:08.674953 2187 ts_tablet_manager.cc:1918] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:30:08.675062 2187 log.cc:1196] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/73d48ab7f63f488bb89c5f7890919a5d
I20260809 09:30:08.675380 2187 ts_tablet_manager.cc:1939] T 73d48ab7f63f488bb89c5f7890919a5d P 4f72ef30dac5468493c7e47c28e0db1c: Deleting consensus metadata
I20260809 09:30:08.676517 1271 catalog_manager.cc:4841] TS 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373): tablet 73d48ab7f63f488bb89c5f7890919a5d (table test-table [id=6e7560c6d30b4397a15e73c0dcc241fc]) successfully deleted
I20260809 09:30:08.676558 2189 ts_tablet_manager.cc:1918] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:30:08.676803 2189 log.cc:1196] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/73d48ab7f63f488bb89c5f7890919a5d
I20260809 09:30:08.677610 2189 ts_tablet_manager.cc:1939] T 73d48ab7f63f488bb89c5f7890919a5d P 9f0e48d0c8e246aea9a13ddd0b972217: Deleting consensus metadata
I20260809 09:30:08.678964 1269 catalog_manager.cc:4841] TS 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087): tablet 73d48ab7f63f488bb89c5f7890919a5d (table test-table [id=6e7560c6d30b4397a15e73c0dcc241fc]) successfully deleted
W20260809 09:30:08.687423 1269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:43639 (error 108)
I20260809 09:30:08.687750 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 1662
I20260809 09:30:08.689854 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:08.689436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:43639 (local address 127.0.0.1:39804)
0809 09:30:08.689657 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:08.689747 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:43639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260809 09:30:08.690083 1269 catalog_manager.cc:4584] TS a7ba480de48747e99a964e8f9a50f440 (127.9.102.1:43639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73d48ab7f63f488bb89c5f7890919a5d: Network error: Client connection negotiation failed: client connection to 127.9.102.1:43639: connect: Connection refused (error 111)
I20260809 09:30:08.699222 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 1889
I20260809 09:30:08.709808 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 1237
I20260809 09:30:08.721583 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40807
--webserver_interface=127.9.102.62
--webserver_port=35297
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:40807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:08.825052 2192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:08.825309 2192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:08.825412 2192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:08.829413 2192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:30:08.829499 2192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:08.829535 2192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:08.829573 2192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:30:08.829608 2192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:30:08.834559 2192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:40807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=35297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:08.835472 2192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:08.836396 2192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:08.843993 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:08.843990 2205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:08.843990 2202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:08.844195 2192 server_base.cc:1034] running on GCE node
I20260809 09:30:08.845005 2192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:08.845997 2192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:08.847187 2192 hybrid_clock.cc:648] HybridClock initialized: now 1786267808847163 us; error 43 us; skew 500 ppm
Aug 09 09:30:08 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267808, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:08.849781 2192 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:30:08.850590 2192 webserver.cc:460] Webserver started at http://127.9.102.62:35297/ using document root <none> and password file <none>
I20260809 09:30:08.851151 2192 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:08.851246 2192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:08.854652 2192 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:08.856734 2223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:08.857815 2192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:08.857954 2192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "d77cb80bc89c4cd2baea34c10760dc76"
format_stamp: "Formatted at 2026-08-09 09:30:06 on dist-test-slave-7vjv"
server_key: "1de7dfecd8991203b253d0de0b3b2755"
server_key_iv: "3f46d4e2f1eee7d06658656552b924dc"
server_key_version: "encryptionkey@0"
I20260809 09:30:08.858408 2192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:08.871508 2192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:08.874135 2192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:08.874322 2192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:08.880996 2192 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:40807
I20260809 09:30:08.881242 2328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:40807 every 8 connection(s)
I20260809 09:30:08.881982 2192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:30:08.885879 2330 sys_catalog.cc:263] Verifying existing consensus state
I20260809 09:30:08.887101 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 2192
I20260809 09:30:08.887509 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:43639
--local_ip_for_outbound_sockets=127.9.102.1
--tserver_master_addrs=127.9.102.62:40807
--webserver_port=38097
--webserver_interface=127.9.102.1
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:30:08.887528 2330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Bootstrap starting.
I20260809 09:30:08.903784 2330 log.cc:824] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:08.914229 2330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 09:30:08.914572 2330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76: Bootstrap complete.
I20260809 09:30:08.918256 2330 raft_consensus.cc:348] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } }
I20260809 09:30:08.918622 2330 raft_consensus.cc:729] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77cb80bc89c4cd2baea34c10760dc76, State: Initialized, Role: FOLLOWER
I20260809 09:30:08.919059 2330 consensus_queue.cc:260] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } }
I20260809 09:30:08.919195 2330 raft_consensus.cc:388] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:30:08.919277 2330 raft_consensus.cc:482] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:30:08.919394 2330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 2 FOLLOWER]: Advancing to term 3
I20260809 09:30:08.921126 2330 raft_consensus.cc:504] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } }
I20260809 09:30:08.921427 2330 leader_election.cc:302] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d77cb80bc89c4cd2baea34c10760dc76; no voters:
I20260809 09:30:08.921696 2330 leader_election.cc:288] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 09:30:08.921823 2335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 09:30:08.922083 2335 raft_consensus.cc:686] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [term 3 LEADER]: Becoming Leader. State: Replica: d77cb80bc89c4cd2baea34c10760dc76, State: Running, Role: LEADER
I20260809 09:30:08.922443 2335 consensus_queue.cc:237] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } }
I20260809 09:30:08.922890 2330 sys_catalog.cc:564] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:30:08.923699 2340 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d77cb80bc89c4cd2baea34c10760dc76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } } }
I20260809 09:30:08.923820 2340 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:08.924155 2342 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d77cb80bc89c4cd2baea34c10760dc76. Latest consensus state: current_term: 3 leader_uuid: "d77cb80bc89c4cd2baea34c10760dc76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d77cb80bc89c4cd2baea34c10760dc76" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40807 } } }
I20260809 09:30:08.924242 2342 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77cb80bc89c4cd2baea34c10760dc76 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:08.924525 2354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:30:08.929299 2354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:30:08.929920 2354 catalog_manager.cc:1259] Loaded cluster ID: d67cfc51a1c64a27872e89584dfb15de
I20260809 09:30:08.930088 2354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 09:30:08.931272 2354 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa3a5d199ad google::LogMessage::Fail() at ??:0
@ 0x7fa3a5d1c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa3a5d194de google::LogMessage::Flush() at ??:0
@ 0x7fa3a5d1aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa3b065cfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa3b065dd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa3b065a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa3b069e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa3a7f6b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa3a7f6bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa3a7f6d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa3a7f5c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa3a95a06db start_thread at ??:0
@ 0x7fa3a481761f clone at ??:0
W20260809 09:30:09.021894 2332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:09.022157 2332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:09.022197 2332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:09.025830 2332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:09.025916 2332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:09.026012 2332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:30:09.030764 2332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:43639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=38097
--tserver_master_addrs=127.9.102.62:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:09.031687 2332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:09.032662 2332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:09.042092 2405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:09.042188 2399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:09.042099 2408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:09.042634 2332 server_base.cc:1034] running on GCE node
I20260809 09:30:09.043047 2332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:09.043756 2332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:09.044960 2332 hybrid_clock.cc:648] HybridClock initialized: now 1786267809044933 us; error 39 us; skew 500 ppm
Aug 09 09:30:09 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267809, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:09.048367 2332 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:30:09.049346 2332 webserver.cc:460] Webserver started at http://127.9.102.1:38097/ using document root <none> and password file <none>
I20260809 09:30:09.049953 2332 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:09.050047 2332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:09.054617 2332 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 09:30:09.057281 2422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:09.058790 2332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:30:09.058912 2332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "a7ba480de48747e99a964e8f9a50f440"
format_stamp: "Formatted at 2026-08-09 09:30:06 on dist-test-slave-7vjv"
server_key: "61cb33e56884fc6d8b350d2b008fccdb"
server_key_iv: "c6e541cb9e5b884c94c7cc4bb2390427"
server_key_version: "encryptionkey@0"
I20260809 09:30:09.059566 2332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:09.072991 2332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:09.076227 2332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:09.076392 2332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:09.077000 2332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:09.078162 2435 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 09:30:09.081892 2332 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 09:30:09.081971 2332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260809 09:30:09.082062 2332 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 09:30:09.084230 2332 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 09:30:09.084285 2332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:09.084605 2435 tablet_bootstrap.cc:492] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440: Bootstrap starting.
I20260809 09:30:09.094573 2332 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:43639
I20260809 09:30:09.094731 2622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:43639 every 8 connection(s)
I20260809 09:30:09.095579 2332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:30:09.096964 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 2332
I20260809 09:30:09.097525 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:39087
--local_ip_for_outbound_sockets=127.9.102.2
--tserver_master_addrs=127.9.102.62:40807
--webserver_port=34289
--webserver_interface=127.9.102.2
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:30:09.105633 2435 log.cc:824] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:09.113691 2435 tablet_bootstrap.cc:492] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 09:30:09.114075 2435 tablet_bootstrap.cc:492] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440: Bootstrap complete.
I20260809 09:30:09.115414 2435 ts_tablet_manager.cc:1397] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.012s
I20260809 09:30:09.118182 2435 raft_consensus.cc:348] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:09.118691 2435 raft_consensus.cc:729] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ba480de48747e99a964e8f9a50f440, State: Initialized, Role: FOLLOWER
I20260809 09:30:09.119199 2435 consensus_queue.cc:260] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:09.120213 2435 ts_tablet_manager.cc:1428] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
W20260809 09:30:09.141750 2629 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:30:09.098742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:57143)
0809 09:30:09.099315 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:09.099392 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:09.100567 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:09.141082 (+ 40515us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":151,"threads_started":1}
W20260809 09:30:09.142185 2625 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:40807 (0 consecutive failures): Network error: Failed to ping master at 127.9.102.62:40807: Client connection negotiation failed: client connection to 127.9.102.62:40807: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 09:30:09.142807 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:09.142569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:43707)
0809 09:30:09.142684 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:09.142761 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 09:30:09.143363 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:09.143173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:34809)
0809 09:30:09.143268 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:09.143310 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 09:30:09.143513 2625 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 09:30:09.207907 2634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:09.208154 2634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:09.208269 2634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:09.211838 2634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:09.211910 2634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:09.211995 2634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:30:09.216843 2634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:39087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=34289
--tserver_master_addrs=127.9.102.62:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:09.217679 2634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:09.218546 2634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:09.226972 2643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:09.226981 2644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:09.226982 2647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:09.227236 2634 server_base.cc:1034] running on GCE node
I20260809 09:30:09.227717 2634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:09.228312 2634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:09.229504 2634 hybrid_clock.cc:648] HybridClock initialized: now 1786267809229493 us; error 29 us; skew 500 ppm
Aug 09 09:30:09 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267809, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:09.231891 2634 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:30:09.232637 2634 webserver.cc:460] Webserver started at http://127.9.102.2:34289/ using document root <none> and password file <none>
I20260809 09:30:09.233093 2634 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:09.233148 2634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:09.236477 2634 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260809 09:30:09.238240 2659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:09.239289 2634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:09.239398 2634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "9f0e48d0c8e246aea9a13ddd0b972217"
format_stamp: "Formatted at 2026-08-09 09:30:07 on dist-test-slave-7vjv"
server_key: "5dddd06c6c393a9c1a1d2ca956cd2be1"
server_key_iv: "79b8d7ce1a6d8f80133c42f316f69d01"
server_key_version: "encryptionkey@0"
I20260809 09:30:09.239866 2634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:09.267966 2634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:09.270247 2634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:09.270407 2634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:09.270872 2634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:09.271538 2634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:30:09.271579 2634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:09.271636 2634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:30:09.271672 2634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:09.279387 2634 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:39087
I20260809 09:30:09.279409 2849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:39087 every 8 connection(s)
I20260809 09:30:09.280197 2634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:30:09.283380 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:09.281848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:34949)
0809 09:30:09.282220 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:09.282306 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
W20260809 09:30:09.283808 2850 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:40807 (0 consecutive failures): Network error: Failed to ping master at 127.9.102.62:40807: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
I20260809 09:30:09.284318 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:09.284141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:54509)
0809 09:30:09.284232 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:09.284273 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 09:30:09.284796 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:09.284639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:43469)
0809 09:30:09.284722 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:09.284757 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 09:30:09.284942 2850 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 09:30:09.286549 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 2634
I20260809 09:30:09.287055 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:41373
--local_ip_for_outbound_sockets=127.9.102.3
--tserver_master_addrs=127.9.102.62:40807
--webserver_port=38765
--webserver_interface=127.9.102.3
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:35115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:09.386606 2858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:09.386886 2858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:09.386967 2858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:09.391145 2858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:09.391191 2858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:09.391239 2858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:30:09.395725 2858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:35115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:41373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=38765
--tserver_master_addrs=127.9.102.62:40807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:09.396489 2858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:09.397291 2858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:09.405521 2868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:09.405540 2864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:09.405586 2858 server_base.cc:1034] running on GCE node
W20260809 09:30:09.405611 2865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:09.406224 2858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:09.406692 2858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:09.407845 2858 hybrid_clock.cc:648] HybridClock initialized: now 1786267809407830 us; error 27 us; skew 500 ppm
Aug 09 09:30:09 dist-test-slave-7vjv krb5kdc[32739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267809, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:09.410496 2858 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:30:09.411257 2858 webserver.cc:460] Webserver started at http://127.9.102.3:38765/ using document root <none> and password file <none>
I20260809 09:30:09.411757 2858 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:09.411813 2858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:09.415055 2858 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:09.416886 2877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:09.417981 2858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:09.418103 2858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "4f72ef30dac5468493c7e47c28e0db1c"
format_stamp: "Formatted at 2026-08-09 09:30:07 on dist-test-slave-7vjv"
server_key: "bb5277e2a448d063d3cd0074b6c18b71"
server_key_iv: "c9003a26966708592c358093f97183f7"
server_key_version: "encryptionkey@0"
I20260809 09:30:09.418548 2858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:09.428910 2858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:09.431190 2858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:09.431366 2858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:09.431857 2858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:09.432611 2858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:30:09.432680 2858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:09.432739 2858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:30:09.432780 2858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:09.440932 2858 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:41373
I20260809 09:30:09.440963 3075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:41373 every 8 connection(s)
I20260809 09:30:09.441807 2858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:30:09.443338 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 2858
I20260809 09:30:09.445494 3080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:09.443831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:59567)
0809 09:30:09.444727 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:09.444840 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":526,"threads_started":1}
W20260809 09:30:09.445955 3076 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:40807 (0 consecutive failures): Network error: Failed to ping master at 127.9.102.62:40807: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
I20260809 09:30:09.446527 3080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:09.446328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:51815)
0809 09:30:09.446428 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:09.446472 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 09:30:09.447160 3080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:09.446963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:54501)
0809 09:30:09.447071 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:09.447111 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 09:30:09.447322 3076 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 09:30:10.144433 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:10.144022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:39007)
0809 09:30:10.144301 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:10.144349 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":106,"threads_started":1}
I20260809 09:30:10.285766 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:10.285420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:47691)
0809 09:30:10.285616 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:10.285662 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":62,"threads_started":1}
I20260809 09:30:10.448195 3084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:10.447811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:55471)
0809 09:30:10.448013 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:10.448071 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
I20260809 09:30:10.863930 3085 raft_consensus.cc:482] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:30:10.864216 3085 raft_consensus.cc:504] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:10.865335 3085 leader_election.cc:288] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373), 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087)
Aug 09 09:30:10 dist-test-slave-7vjv krb5kdc[32739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267809, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.3@KRBTEST.COM
Aug 09 09:30:10 dist-test-slave-7vjv krb5kdc[32739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267809, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.2@KRBTEST.COM
I20260809 09:30:10.880458 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:10.865508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:41373 (local address 127.9.102.1:45425)
0809 09:30:10.865891 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:10.865900 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:10.865952 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:10.867804 (+ 1852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:10.867810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:10.868439 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:10.869204 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:10.869224 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:10.871187 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:10.871191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:10.872620 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:10.872631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:10.873510 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:10.873513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:10.873767 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:10.874598 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:10.874624 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:10.876301 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:10.878218 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:10.878223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:10.878234 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:10.878453 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:10.878776 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:10.878779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:10.878781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:10.878946 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:10.879684 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:10.879688 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:10.879866 (+ 178us) client_negotiation.cc:769] Sending connection context
0809 09:30:10.880004 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 09:30:10.880295 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":372,"spinlock_wait_cycles":19968,"thread_start_us":51,"threads_started":1}
I20260809 09:30:10.880605 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:10.865508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:39087 (local address 127.9.102.1:44653)
0809 09:30:10.865816 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:10.865830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:10.865946 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:10.867804 (+ 1858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:10.867810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:10.868439 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:10.869204 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:10.869224 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:10.871187 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:10.871191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:10.872620 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:10.872631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:10.873510 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:10.873513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:10.873769 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:10.874861 (+ 1092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:10.874872 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:10.877064 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:10.878886 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:10.878889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:10.878890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:10.879095 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:10.879417 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:10.879419 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:10.879420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:10.879480 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:10.880132 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:10.880134 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:10.880226 (+ 92us) client_negotiation.cc:769] Sending connection context
0809 09:30:10.880332 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 09:30:10.880426 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":218,"spinlock_wait_cycles":1280,"thread_start_us":87,"threads_started":1}
I20260809 09:30:10.880852 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:10.865683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:45425 (local address 127.9.102.3:41373)
0809 09:30:10.866073 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:10.866092 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:10.866131 (+ 39us) server_negotiation.cc:407] Connection header received
0809 09:30:10.866976 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:10.866983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:10.867392 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:10.867704 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:10.869540 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:10.870952 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:10.872737 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:10.873392 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:10.873560 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:10.876527 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:10.876541 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:10.876551 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:10.876576 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:10.878108 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:10.878582 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:10.878587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:10.878590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:10.878687 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:10.879052 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:10.879055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:10.879056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:10.879561 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:10.879712 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:10.880226 (+ 514us) server_negotiation.cc:299] Negotiation successful
0809 09:30:10.880503 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":104,"threads_started":1}
I20260809 09:30:10.880877 3092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:10.865683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:44653 (local address 127.9.102.2:39087)
0809 09:30:10.866073 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:10.866088 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:10.866131 (+ 43us) server_negotiation.cc:407] Connection header received
0809 09:30:10.866976 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:10.866983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:10.867391 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:10.867704 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:10.869552 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:10.870952 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:10.872737 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:10.873392 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:10.873568 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:10.877230 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:10.877234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:10.877244 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:10.877268 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:10.878797 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:10.879203 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:10.879208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:10.879212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:10.879332 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:10.879575 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:10.879578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:10.879579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:10.880004 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:10.880137 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:10.880480 (+ 343us) server_negotiation.cc:299] Negotiation successful
0809 09:30:10.880743 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":116,"threads_started":1}
W20260809 09:30:10.883116 2426 leader_election.cc:341] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373): Not found: Tablet not found: 73d48ab7f63f488bb89c5f7890919a5d
W20260809 09:30:10.883296 2424 leader_election.cc:341] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087): Not found: Tablet not found: 73d48ab7f63f488bb89c5f7890919a5d
I20260809 09:30:10.883352 2424 leader_election.cc:302] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a7ba480de48747e99a964e8f9a50f440; no voters: 4f72ef30dac5468493c7e47c28e0db1c, 9f0e48d0c8e246aea9a13ddd0b972217
I20260809 09:30:10.883618 3085 raft_consensus.cc:2726] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 09:30:11.145321 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:11.145067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:34145)
0809 09:30:11.145186 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:11.145244 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 09:30:11.286891 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:11.286499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:45231)
0809 09:30:11.286747 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:11.286797 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
I20260809 09:30:11.449349 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:11.448996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:51405)
0809 09:30:11.449227 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:11.449278 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260809 09:30:12.146508 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:12.146126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:36245)
0809 09:30:12.146368 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:12.146423 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260809 09:30:12.287930 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:12.287554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:52479)
0809 09:30:12.287786 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:12.287848 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260809 09:30:12.450510 3103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:12.450152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:55817)
0809 09:30:12.450382 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:12.450435 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260809 09:30:12.521138 3105 raft_consensus.cc:482] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:30:12.521289 3105 raft_consensus.cc:504] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:12.521816 3105 leader_election.cc:288] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373), 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087)
W20260809 09:30:12.522644 2426 leader_election.cc:341] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373): Not found: Tablet not found: 73d48ab7f63f488bb89c5f7890919a5d
W20260809 09:30:12.522738 2424 leader_election.cc:341] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087): Not found: Tablet not found: 73d48ab7f63f488bb89c5f7890919a5d
I20260809 09:30:12.522791 2424 leader_election.cc:302] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a7ba480de48747e99a964e8f9a50f440; no voters: 4f72ef30dac5468493c7e47c28e0db1c, 9f0e48d0c8e246aea9a13ddd0b972217
I20260809 09:30:12.522975 3105 raft_consensus.cc:2726] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 09:30:13.147702 3106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:13.147251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:34757)
0809 09:30:13.147555 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:13.147614 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
I20260809 09:30:13.289144 3107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:13.288777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:55151)
0809 09:30:13.289018 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:13.289071 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":60,"threads_started":1}
I20260809 09:30:13.451444 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:13.451131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:55483)
0809 09:30:13.451329 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:13.451372 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":62,"threads_started":1}
I20260809 09:30:14.148983 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:14.148545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:44231)
0809 09:30:14.148820 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:14.148871 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":70,"threads_started":1}
I20260809 09:30:14.290179 3115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:14.289783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:50185)
0809 09:30:14.290019 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:14.290094 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":65,"threads_started":1}
I20260809 09:30:14.452538 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:14.452194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:58317)
0809 09:30:14.452410 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:14.452459 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":61,"threads_started":1}
I20260809 09:30:15.149880 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:15.149559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:33147)
0809 09:30:15.149760 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:15.149802 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260809 09:30:15.291344 3119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:15.290997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:35859)
0809 09:30:15.291225 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:15.291272 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
I20260809 09:30:15.453547 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:15.453207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:35369)
0809 09:30:15.453421 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:15.453469 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":64,"threads_started":1}
I20260809 09:30:15.622931 3122 raft_consensus.cc:482] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:30:15.623059 3122 raft_consensus.cc:504] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:15.623441 3122 leader_election.cc:288] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373), 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087)
W20260809 09:30:15.624203 2426 leader_election.cc:341] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373): Not found: Tablet not found: 73d48ab7f63f488bb89c5f7890919a5d
W20260809 09:30:15.624306 2424 leader_election.cc:341] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087): Not found: Tablet not found: 73d48ab7f63f488bb89c5f7890919a5d
I20260809 09:30:15.624370 2424 leader_election.cc:302] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a7ba480de48747e99a964e8f9a50f440; no voters: 4f72ef30dac5468493c7e47c28e0db1c, 9f0e48d0c8e246aea9a13ddd0b972217
I20260809 09:30:15.624497 3122 raft_consensus.cc:2726] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 09:30:16.151108 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:16.150738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:57421)
0809 09:30:16.150952 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:16.151012 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260809 09:30:16.292608 3126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:16.292213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:33001)
0809 09:30:16.292477 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:16.292524 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":61,"threads_started":1}
I20260809 09:30:16.454772 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:16.454282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:59219)
0809 09:30:16.454650 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:16.454699 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":77,"threads_started":1}
I20260809 09:30:17.152376 3132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:17.151875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:38993)
0809 09:30:17.152227 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:17.152276 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":64,"threads_started":1}
I20260809 09:30:17.293639 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:17.293244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:49517)
0809 09:30:17.293504 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:17.293553 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":104,"threads_started":1}
I20260809 09:30:17.455735 3134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:17.455398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:46357)
0809 09:30:17.455617 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:17.455662 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260809 09:30:18.153409 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:18.153055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:57445)
0809 09:30:18.153274 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:18.153323 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260809 09:30:18.294624 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:18.294275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:45777)
0809 09:30:18.294491 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:18.294546 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
I20260809 09:30:18.456784 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:18.456440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:42561)
0809 09:30:18.456657 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:18.456706 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260809 09:30:19.154582 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:19.154232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:57321)
0809 09:30:19.154445 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:19.154491 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260809 09:30:19.295743 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:19.295408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:51961)
0809 09:30:19.295628 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:19.295674 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":65,"threads_started":1}
I20260809 09:30:19.409188 3152 raft_consensus.cc:482] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:30:19.409317 3152 raft_consensus.cc:504] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:19.409804 3152 leader_election.cc:288] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373), 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087)
W20260809 09:30:19.410553 2426 leader_election.cc:341] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373): Not found: Tablet not found: 73d48ab7f63f488bb89c5f7890919a5d
W20260809 09:30:19.410676 2424 leader_election.cc:341] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087): Not found: Tablet not found: 73d48ab7f63f488bb89c5f7890919a5d
I20260809 09:30:19.410777 2424 leader_election.cc:302] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a7ba480de48747e99a964e8f9a50f440; no voters: 4f72ef30dac5468493c7e47c28e0db1c, 9f0e48d0c8e246aea9a13ddd0b972217
I20260809 09:30:19.410919 3152 raft_consensus.cc:2726] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 09:30:19.457710 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:19.457381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:45251)
0809 09:30:19.457589 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:19.457637 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
I20260809 09:30:20.155578 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:20.155229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:41183)
0809 09:30:20.155443 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:20.155492 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260809 09:30:20.296810 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:20.296420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:50607)
0809 09:30:20.296655 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:20.296721 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
I20260809 09:30:20.458767 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:20.458340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:44881)
0809 09:30:20.458633 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:20.458687 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":65,"threads_started":1}
I20260809 09:30:21.156657 3162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:21.156253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:36511)
0809 09:30:21.156500 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:21.156555 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
I20260809 09:30:21.297752 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:21.297392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:46265)
0809 09:30:21.297622 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:21.297671 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":78,"threads_started":1}
I20260809 09:30:21.459805 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:21.459455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:56271)
0809 09:30:21.459677 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:21.459726 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
I20260809 09:30:21.682618 3166 raft_consensus.cc:482] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:30:21.682736 3166 raft_consensus.cc:504] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f72ef30dac5468493c7e47c28e0db1c" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 41373 } } peers { permanent_uuid: "9f0e48d0c8e246aea9a13ddd0b972217" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 39087 } } peers { permanent_uuid: "a7ba480de48747e99a964e8f9a50f440" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43639 } }
I20260809 09:30:21.683115 3166 leader_election.cc:288] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373), 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087)
W20260809 09:30:21.683810 2426 leader_election.cc:341] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4f72ef30dac5468493c7e47c28e0db1c (127.9.102.3:41373): Not found: Tablet not found: 73d48ab7f63f488bb89c5f7890919a5d
W20260809 09:30:21.683910 2424 leader_election.cc:341] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f0e48d0c8e246aea9a13ddd0b972217 (127.9.102.2:39087): Not found: Tablet not found: 73d48ab7f63f488bb89c5f7890919a5d
I20260809 09:30:21.683997 2424 leader_election.cc:302] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a7ba480de48747e99a964e8f9a50f440; no voters: 4f72ef30dac5468493c7e47c28e0db1c, 9f0e48d0c8e246aea9a13ddd0b972217
I20260809 09:30:21.684154 3166 raft_consensus.cc:2726] T 73d48ab7f63f488bb89c5f7890919a5d P a7ba480de48747e99a964e8f9a50f440 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 09:30:22.157636 3167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:22.157292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:33819)
0809 09:30:22.157497 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:22.157550 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260809 09:30:22.298786 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:22.298452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:48705)
0809 09:30:22.298667 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:22.298713 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":77,"threads_started":1}
I20260809 09:30:22.460783 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:22.460447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:46657)
0809 09:30:22.460660 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:22.460707 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260809 09:30:23.158625 3170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:23.158273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:44721)
0809 09:30:23.158494 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:23.158542 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
I20260809 09:30:23.299856 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:23.299493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:57007)
0809 09:30:23.299736 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:23.299783 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260809 09:30:23.461725 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:23.461367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:41719)
0809 09:30:23.461576 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:23.461642 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260809 09:30:24.159817 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:24.159458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.1:57703)
0809 09:30:24.159687 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:24.159733 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
I20260809 09:30:24.300940 3176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:24.300606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.2:60367)
0809 09:30:24.300819 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:24.300864 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
I20260809 09:30:24.453609 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 2332
I20260809 09:30:24.463812 3181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:24.463386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40807 (local address 127.9.102.3:48809)
0809 09:30:24.463631 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:24.463694 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":62,"threads_started":1}
I20260809 09:30:24.470213 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 2634
I20260809 09:30:24.479705 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 2858
2026-08-09T09:30:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19147 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:30:24 dist-test-slave-7vjv krb5kdc[3187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:30:24 dist-test-slave-7vjv krb5kdc[3187](info): set up 2 sockets
Aug 09 09:30:24 dist-test-slave-7vjv krb5kdc[3187](info): commencing operation
krb5kdc: starting...
W20260809 09:30:25.648365 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:30:25 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:30:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:30:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:30:25.756938 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40855
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:40855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:25.853369 3203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:25.853579 3203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:25.853647 3203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:25.856954 3203 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 09:30:25.857026 3203 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 09:30:25.857052 3203 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 09:30:25.857074 3203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:30:25.857122 3203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:25.857160 3203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:25.857194 3203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:30:25.857226 3203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:30:25.861789 3203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:40855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:25.862551 3203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:25.863324 3203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 09:30:25.870733 3203 server_base.cc:1034] running on GCE node
W20260809 09:30:25.870801 3216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:25.870764 3210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:25.870774 3212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:25.871471 3203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:25.872460 3203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:25.873612 3203 hybrid_clock.cc:648] HybridClock initialized: now 1786267825873591 us; error 33 us; skew 500 ppm
Aug 09 09:30:25 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267825, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:25.875928 3203 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:30:25.876703 3203 webserver.cc:460] Webserver started at http://127.9.102.62:46207/ using document root <none> and password file <none>
I20260809 09:30:25.877149 3203 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:25.877226 3203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:25.877434 3203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:30:25.879158 3203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "24691abac8754f36bd7a6146d48893d3"
format_stamp: "Formatted at 2026-08-09 09:30:25 on dist-test-slave-7vjv"
server_key: "17be5d4af0a956a902d4e039671fb4fc"
server_key_iv: "a706a46c360a3f5d74b748b4c5b0e0ce"
server_key_version: "encryptionkey@0"
I20260809 09:30:25.879664 3203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "24691abac8754f36bd7a6146d48893d3"
format_stamp: "Formatted at 2026-08-09 09:30:25 on dist-test-slave-7vjv"
server_key: "17be5d4af0a956a902d4e039671fb4fc"
server_key_iv: "a706a46c360a3f5d74b748b4c5b0e0ce"
server_key_version: "encryptionkey@0"
I20260809 09:30:25.882771 3203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:30:25.884812 3235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:25.885825 3203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:25.885942 3203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "24691abac8754f36bd7a6146d48893d3"
format_stamp: "Formatted at 2026-08-09 09:30:25 on dist-test-slave-7vjv"
server_key: "17be5d4af0a956a902d4e039671fb4fc"
server_key_iv: "a706a46c360a3f5d74b748b4c5b0e0ce"
server_key_version: "encryptionkey@0"
I20260809 09:30:25.886031 3203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:25.920421 3203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:25.922780 3203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:25.922955 3203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:25.929391 3203 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:40855
I20260809 09:30:25.929661 3331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:40855 every 8 connection(s)
I20260809 09:30:25.930330 3203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:30:25.932952 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 3203
I20260809 09:30:25.933084 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:30:25.933097 3332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:25.933420 9624 external_mini_cluster.cc:1428] Setting key 3d947760da837c8328feca134d359ed6
I20260809 09:30:25.939674 3332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Bootstrap starting.
Aug 09 09:30:25 dist-test-slave-7vjv krb5kdc[3187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:25.942137 3332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:25.943017 3332 log.cc:824] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:25.944972 3332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: No bootstrap required, opened a new log
I20260809 09:30:25.946627 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:25.934625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.9.102.62:40855)
0809 09:30:25.934983 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:25.934997 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:25.935027 (+ 30us) server_negotiation.cc:407] Connection header received
0809 09:30:25.935576 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:25.935590 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:25.935887 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:25.936172 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:25.937041 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:25.937992 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:25.939121 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:25.939500 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:25.939618 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:25.942104 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:25.942112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:25.942127 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:25.942157 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:25.943776 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:25.944214 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:25.944219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:25.944224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:25.944352 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:25.944604 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:25.944607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:25.944608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:25.945046 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:25.945212 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:25.945476 (+ 264us) server_negotiation.cc:299] Negotiation successful
0809 09:30:25.945767 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":126,"threads_started":1}
I20260809 09:30:25.948976 3332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } }
I20260809 09:30:25.949167 3332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:25.949208 3332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24691abac8754f36bd7a6146d48893d3, State: Initialized, Role: FOLLOWER
I20260809 09:30:25.949517 3332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } }
I20260809 09:30:25.949620 3332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:30:25.949659 3332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:30:25.949720 3332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:25.950412 3332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } }
I20260809 09:30:25.950712 3332 leader_election.cc:302] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24691abac8754f36bd7a6146d48893d3; no voters:
I20260809 09:30:25.950943 3332 leader_election.cc:288] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:30:25.951006 3338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:30:25.951218 3338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 1 LEADER]: Becoming Leader. State: Replica: 24691abac8754f36bd7a6146d48893d3, State: Running, Role: LEADER
I20260809 09:30:25.951479 3338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } }
I20260809 09:30:25.952086 3332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:30:25.952665 3340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24691abac8754f36bd7a6146d48893d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } } }
I20260809 09:30:25.952803 3340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:25.953169 3339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24691abac8754f36bd7a6146d48893d3. Latest consensus state: current_term: 1 leader_uuid: "24691abac8754f36bd7a6146d48893d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } } }
I20260809 09:30:25.953266 3346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:30:25.953343 3339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:25.955698 3346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:30:25.960350 3346 catalog_manager.cc:1347] Generated new cluster ID: 61aead23013f41c9b608f3969d91ece0
I20260809 09:30:25.960402 3346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:30:25.976059 3346 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:30:25.976822 3346 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:30:25.984316 3346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Generated new TSK 0
I20260809 09:30:25.984827 3346 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:30:26.015103 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40855
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:26.115152 3366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:26.115343 3366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:26.115377 3366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:26.118755 3366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:26.118799 3366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:26.118863 3366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:30:26.123246 3366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:26.124048 3366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:26.124969 3366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:26.132750 3385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:26.132807 3381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:26.132850 3366 server_base.cc:1034] running on GCE node
W20260809 09:30:26.132794 3380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:26.133272 3366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:26.133818 3366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:26.134995 3366 hybrid_clock.cc:648] HybridClock initialized: now 1786267826134983 us; error 27 us; skew 500 ppm
Aug 09 09:30:26 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267826, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:26.137377 3366 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:30:26.138093 3366 webserver.cc:460] Webserver started at http://127.9.102.1:42599/ using document root <none> and password file <none>
I20260809 09:30:26.138581 3366 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:26.138659 3366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:26.138877 3366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:30:26.140561 3366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "e1737b2a81854cf0ad1384f1e7538922"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "24eef352574eab1027b6b62646d1a9d9"
server_key_iv: "9d7c5b318c1f8738cfbc7fb93bbc3d07"
server_key_version: "encryptionkey@0"
I20260809 09:30:26.141047 3366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "e1737b2a81854cf0ad1384f1e7538922"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "24eef352574eab1027b6b62646d1a9d9"
server_key_iv: "9d7c5b318c1f8738cfbc7fb93bbc3d07"
server_key_version: "encryptionkey@0"
I20260809 09:30:26.144034 3366 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:30:26.146152 3398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:26.147115 3366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 09:30:26.147230 3366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "e1737b2a81854cf0ad1384f1e7538922"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "24eef352574eab1027b6b62646d1a9d9"
server_key_iv: "9d7c5b318c1f8738cfbc7fb93bbc3d07"
server_key_version: "encryptionkey@0"
I20260809 09:30:26.147321 3366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:26.157009 3366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:26.159260 3366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:26.159421 3366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:26.159922 3366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:26.160663 3366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:30:26.160727 3366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:26.160789 3366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:30:26.160825 3366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:26.168473 3366 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:38573
I20260809 09:30:26.168495 3595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:38573 every 8 connection(s)
I20260809 09:30:26.169163 3366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:30:26.170482 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 3366
I20260809 09:30:26.170610 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:30:26.170908 9624 external_mini_cluster.cc:1428] Setting key 0ec4d9787d64813a0d9c9c0c6cfb83f3
Aug 09 09:30:26 dist-test-slave-7vjv krb5kdc[3187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267826, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:30:26.181772 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.170647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:35781 (local address 127.9.102.62:40855)
0809 09:30:26.170787 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:26.170792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:26.171600 (+ 808us) server_negotiation.cc:407] Connection header received
0809 09:30:26.172947 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:26.172950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:26.172989 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:26.173062 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:26.174448 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.174929 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.175909 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.176177 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.176284 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.178448 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:26.178451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:26.178453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:26.178474 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:26.179824 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:26.180364 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:26.180367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:26.180368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:26.180444 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:26.180756 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:26.180760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:26.180761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:26.180926 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:26.181023 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:26.181469 (+ 446us) server_negotiation.cc:299] Negotiation successful
0809 09:30:26.181599 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 09:30:26.182503 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.170983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:35781)
0809 09:30:26.171421 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:26.171482 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:26.172746 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:26.173213 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:26.173223 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:26.173664 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:26.174222 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.174260 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.175059 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.175061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.175807 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.175815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.176318 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.176322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.176468 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.176956 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:26.176970 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:26.178296 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:26.179934 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:26.179940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:26.179951 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:26.180260 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:26.180536 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:26.180539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:26.180540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:26.180667 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:26.181033 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:26.181041 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:26.181263 (+ 222us) client_negotiation.cc:769] Sending connection context
0809 09:30:26.181425 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 09:30:26.181696 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":117,"threads_started":1}
I20260809 09:30:26.183585 3596 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
I20260809 09:30:26.183796 3596 heartbeater.cc:464] Registering TS with master...
I20260809 09:30:26.184329 3596 heartbeater.cc:511] Master 127.9.102.62:40855 requested a full tablet report, sending...
I20260809 09:30:26.185662 3276 ts_manager.cc:194] Registered new tserver with Master: e1737b2a81854cf0ad1384f1e7538922 (127.9.102.1:38573)
I20260809 09:30:26.186643 3276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:35781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:30:26.197311 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40855
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:26.298759 3609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:26.298979 3609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:26.299017 3609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:26.302459 3609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:26.302510 3609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:26.302613 3609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:30:26.307017 3609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:26.307790 3609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:26.308707 3609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:26.316628 3619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:26.316720 3609 server_base.cc:1034] running on GCE node
W20260809 09:30:26.316649 3623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:26.316637 3626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:26.317160 3609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:26.317754 3609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:26.318912 3609 hybrid_clock.cc:648] HybridClock initialized: now 1786267826318902 us; error 30 us; skew 500 ppm
Aug 09 09:30:26 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267826, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:26.321527 3609 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:30:26.322283 3609 webserver.cc:460] Webserver started at http://127.9.102.2:32933/ using document root <none> and password file <none>
I20260809 09:30:26.322788 3609 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:26.322875 3609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:26.323087 3609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:30:26.324872 3609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "9eb88197ab8e47f1979f0b882e18af33"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "190b53b1c8fc7622a603534948d7cdb6"
server_key_iv: "6727519ec2395abd46ce8d04881c6aa1"
server_key_version: "encryptionkey@0"
I20260809 09:30:26.325354 3609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "9eb88197ab8e47f1979f0b882e18af33"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "190b53b1c8fc7622a603534948d7cdb6"
server_key_iv: "6727519ec2395abd46ce8d04881c6aa1"
server_key_version: "encryptionkey@0"
I20260809 09:30:26.328413 3609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 09:30:26.330464 3636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:26.331514 3609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:26.331632 3609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "9eb88197ab8e47f1979f0b882e18af33"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "190b53b1c8fc7622a603534948d7cdb6"
server_key_iv: "6727519ec2395abd46ce8d04881c6aa1"
server_key_version: "encryptionkey@0"
I20260809 09:30:26.331724 3609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:26.352646 3609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:26.355428 3609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:26.355592 3609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:26.356065 3609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:26.356789 3609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:30:26.356851 3609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:26.356910 3609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:30:26.356952 3609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:26.364470 3609 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:34269
I20260809 09:30:26.364501 3836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:34269 every 8 connection(s)
I20260809 09:30:26.365199 3609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:30:26.372638 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 3609
Aug 09 09:30:26 dist-test-slave-7vjv krb5kdc[3187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267826, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:26.372785 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:30:26.373088 9624 external_mini_cluster.cc:1428] Setting key 3321799be2d65c088c29796362fde79c
I20260809 09:30:26.377456 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.366638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:37639 (local address 127.9.102.62:40855)
0809 09:30:26.366781 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:26.366785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:26.367446 (+ 661us) server_negotiation.cc:407] Connection header received
0809 09:30:26.368562 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:26.368565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:26.368621 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:26.368685 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:26.369579 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.370016 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.370945 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.371143 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.371209 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.373344 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:26.373347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:26.373349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:26.373371 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:26.375257 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:26.375820 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:26.375822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:26.375823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:26.375889 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:26.376260 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:26.376263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:26.376268 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:26.376500 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:26.376633 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:26.377161 (+ 528us) server_negotiation.cc:299] Negotiation successful
0809 09:30:26.377299 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 09:30:26.378299 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.366923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:37639)
0809 09:30:26.367296 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:26.367329 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:26.368393 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:26.368822 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:26.368828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:26.369101 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:26.369463 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.369474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.370114 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.370117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.370852 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.370860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.371263 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.371265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.371384 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.371798 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:26.371813 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:26.373194 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:26.375367 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:26.375374 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:26.375390 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:26.375735 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:26.375965 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:26.375970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:26.375974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:26.376161 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:26.376650 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:26.376654 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:26.376929 (+ 275us) client_negotiation.cc:769] Sending connection context
0809 09:30:26.377105 (+ 176us) client_negotiation.cc:240] Negotiation successful
0809 09:30:26.377402 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":102,"threads_started":1}
I20260809 09:30:26.379655 3838 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40855
I20260809 09:30:26.379905 3838 heartbeater.cc:464] Registering TS with master...
I20260809 09:30:26.380434 3838 heartbeater.cc:511] Master 127.9.102.62:40855 requested a full tablet report, sending...
I20260809 09:30:26.381543 3276 ts_manager.cc:194] Registered new tserver with Master: 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269)
I20260809 09:30:26.382385 3276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:37639
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:30:26.404407 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40855
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:26.504551 3845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:26.504752 3845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:26.504788 3845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:26.508176 3845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:26.508219 3845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:26.508267 3845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:30:26.512436 3845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:26.513144 3845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:26.514041 3845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:26.522281 3858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:26.522316 3853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:26.522254 3855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:26.522609 3845 server_base.cc:1034] running on GCE node
I20260809 09:30:26.522924 3845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:26.523451 3845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:26.524613 3845 hybrid_clock.cc:648] HybridClock initialized: now 1786267826524597 us; error 26 us; skew 500 ppm
Aug 09 09:30:26 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267826, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:26.526952 3845 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:30:26.527681 3845 webserver.cc:460] Webserver started at http://127.9.102.3:33889/ using document root <none> and password file <none>
I20260809 09:30:26.528180 3845 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:26.528254 3845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:26.528394 3845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:30:26.530041 3845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "82dd3517f63f4ee2aabdcc6e0716c097"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "bd3c7a4fab525e914c9e8e4661c6cd7a"
server_key_iv: "90d1c3fa1843073a8d575c31f63fdf1c"
server_key_version: "encryptionkey@0"
I20260809 09:30:26.530472 3845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "82dd3517f63f4ee2aabdcc6e0716c097"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "bd3c7a4fab525e914c9e8e4661c6cd7a"
server_key_iv: "90d1c3fa1843073a8d575c31f63fdf1c"
server_key_version: "encryptionkey@0"
I20260809 09:30:26.533587 3845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:30:26.535563 3869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:26.536652 3845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:26.536760 3845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "82dd3517f63f4ee2aabdcc6e0716c097"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "bd3c7a4fab525e914c9e8e4661c6cd7a"
server_key_iv: "90d1c3fa1843073a8d575c31f63fdf1c"
server_key_version: "encryptionkey@0"
I20260809 09:30:26.536849 3845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:26.559124 3845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:26.561810 3845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:26.561932 3845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:26.562326 3845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:26.562944 3845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:30:26.562985 3845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:26.563017 3845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:30:26.563045 3845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:26.571111 3845 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:46267
I20260809 09:30:26.571419 4057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:46267 every 8 connection(s)
I20260809 09:30:26.571950 3845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
Aug 09 09:30:26 dist-test-slave-7vjv krb5kdc[3187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267826, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:26.579787 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 3845
I20260809 09:30:26.579888 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:30:26.580192 9624 external_mini_cluster.cc:1428] Setting key 97165065817874bb66b4a46c4bece750
I20260809 09:30:26.583632 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.573485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:43377 (local address 127.9.102.62:40855)
0809 09:30:26.573608 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:26.573612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:26.574280 (+ 668us) server_negotiation.cc:407] Connection header received
0809 09:30:26.575074 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:26.575077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:26.575134 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:26.575224 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:26.576245 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.576731 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.577726 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.577940 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.578042 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.580206 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:26.580209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:26.580211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:26.580245 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:26.581685 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:26.582260 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:26.582263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:26.582265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:26.582345 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:26.582654 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:26.582658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:26.582659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:26.582823 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:26.582929 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:26.583362 (+ 433us) server_negotiation.cc:299] Negotiation successful
0809 09:30:26.583479 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 09:30:26.584317 4060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.573774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:43377)
0809 09:30:26.574154 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:26.574189 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:26.574887 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:26.575373 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:26.575379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:26.575680 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:26.576053 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.576064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.576844 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.576847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.577608 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.577617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.578068 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.578072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.578234 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.578644 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:26.578658 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:26.580041 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:26.581810 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:26.581815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:26.581827 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:26.582161 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:26.582429 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:26.582432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:26.582433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:26.582564 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:26.582930 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:26.582936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:26.583156 (+ 220us) client_negotiation.cc:769] Sending connection context
0809 09:30:26.583320 (+ 164us) client_negotiation.cc:240] Negotiation successful
0809 09:30:26.583584 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":102,"threads_started":1}
I20260809 09:30:26.585151 4058 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40855
I20260809 09:30:26.585343 4058 heartbeater.cc:464] Registering TS with master...
I20260809 09:30:26.585742 4058 heartbeater.cc:511] Master 127.9.102.62:40855 requested a full tablet report, sending...
I20260809 09:30:26.586575 3276 ts_manager.cc:194] Registered new tserver with Master: 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267)
I20260809 09:30:26.587062 3276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:43377
I20260809 09:30:26.593357 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:30:26.601223 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.595054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.9.102.62:40855)
0809 09:30:26.595150 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:26.595152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:26.595335 (+ 183us) server_negotiation.cc:407] Connection header received
0809 09:30:26.595414 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:26.595416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:26.595445 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:26.595499 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:26.595989 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.596430 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.597351 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.597547 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.597614 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.598772 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:26.598774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:26.598775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:26.598789 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:26.599934 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:26.600351 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:26.600354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:26.600355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:26.600426 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:26.600641 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:26.600643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:26.600644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:26.600817 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:26.600904 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:26.601064 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 09:30:26.601138 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 09:30:26.603727 3276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:30:26.606295 3276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:30:26.618893 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.614086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.9.102.3:46267)
0809 09:30:26.614452 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:26.614463 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:26.614866 (+ 403us) server_negotiation.cc:407] Connection header received
0809 09:30:26.614926 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:26.614930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:26.615113 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:26.615267 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:26.615613 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.616598 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.617810 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.618293 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.618403 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.618523 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:26.618692 (+ 169us) server_negotiation.cc:299] Negotiation successful
0809 09:30:26.618783 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260809 09:30:26.618885 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.613261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38573 (local address 127.0.0.1:51116)
0809 09:30:26.613723 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:26.613764 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:26.613883 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:26.614874 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:26.614880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:26.614901 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:26.615011 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.615017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.616235 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.616238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.617463 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.617474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.618463 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.618466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.618554 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.618574 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 09:30:26.618624 (+ 50us) client_negotiation.cc:240] Negotiation successful
0809 09:30:26.618737 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":55,"thread_start_us":54,"threads_started":1}
I20260809 09:30:26.619117 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.613398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34269 (local address 127.0.0.1:41140)
0809 09:30:26.614247 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:26.614260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:26.614348 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:26.614991 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:26.614994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:26.615028 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:26.615127 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.615133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.616359 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.616362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.617489 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.617498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.618635 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.618638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.618709 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.618726 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 09:30:26.618794 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 09:30:26.618845 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":32,"thread_start_us":101,"threads_started":1}
I20260809 09:30:26.619191 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.613556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.9.102.2:34269)
0809 09:30:26.614171 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:26.614183 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:26.614436 (+ 253us) server_negotiation.cc:407] Connection header received
0809 09:30:26.614503 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:26.614507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:26.614678 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:26.614825 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:26.615259 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.616230 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.617637 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.618329 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.618438 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.618579 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:26.618951 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 09:30:26.619047 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":99,"threads_started":1}
I20260809 09:30:26.618892 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.613964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:46267 (local address 127.0.0.1:46724)
0809 09:30:26.614663 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:26.614675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:26.614761 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:26.615395 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:26.615398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:26.615413 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:26.615494 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.615499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.616728 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.616731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.617712 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.617719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.618408 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.618411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.618503 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.618545 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 09:30:26.618651 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 09:30:26.618762 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":22,"thread_start_us":104,"threads_started":1}
I20260809 09:30:26.619603 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.613807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.9.102.1:38573)
0809 09:30:26.614382 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:26.614394 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:26.614415 (+ 21us) server_negotiation.cc:407] Connection header received
0809 09:30:26.614489 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:26.614494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:26.614678 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:26.614849 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:26.615131 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.616078 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.617581 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.618329 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.619198 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.619332 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:26.619378 (+ 46us) server_negotiation.cc:299] Negotiation successful
0809 09:30:26.619457 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":99,"threads_started":1}
I20260809 09:30:26.621922 3953 tablet_service.cc:1467] Processing CreateTablet for tablet e58aefdcbb9444479e6df8d68dff1c55 (DEFAULT_TABLE table=test-table [id=5412f7a5a3434bf7b8c4c792e8417099]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:30:26.621923 3718 tablet_service.cc:1467] Processing CreateTablet for tablet e58aefdcbb9444479e6df8d68dff1c55 (DEFAULT_TABLE table=test-table [id=5412f7a5a3434bf7b8c4c792e8417099]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:30:26.622072 3487 tablet_service.cc:1467] Processing CreateTablet for tablet e58aefdcbb9444479e6df8d68dff1c55 (DEFAULT_TABLE table=test-table [id=5412f7a5a3434bf7b8c4c792e8417099]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:30:26.622850 3487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58aefdcbb9444479e6df8d68dff1c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:26.622887 3718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58aefdcbb9444479e6df8d68dff1c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:26.622889 3953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58aefdcbb9444479e6df8d68dff1c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:26.629211 4093 tablet_bootstrap.cc:492] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Bootstrap starting.
I20260809 09:30:26.630170 4091 tablet_bootstrap.cc:492] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33: Bootstrap starting.
I20260809 09:30:26.630381 4089 tablet_bootstrap.cc:492] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097: Bootstrap starting.
I20260809 09:30:26.631814 4093 tablet_bootstrap.cc:654] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:26.632043 4091 tablet_bootstrap.cc:654] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:26.632220 4089 tablet_bootstrap.cc:654] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:26.632732 4091 log.cc:824] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:26.632710 4093 log.cc:824] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:26.633267 4089 log.cc:824] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:26.634588 4091 tablet_bootstrap.cc:492] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33: No bootstrap required, opened a new log
I20260809 09:30:26.634815 4091 ts_tablet_manager.cc:1397] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:30:26.635095 4093 tablet_bootstrap.cc:492] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: No bootstrap required, opened a new log
I20260809 09:30:26.635110 4089 tablet_bootstrap.cc:492] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097: No bootstrap required, opened a new log
I20260809 09:30:26.635289 4093 ts_tablet_manager.cc:1397] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 09:30:26.635289 4089 ts_tablet_manager.cc:1397] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260809 09:30:26.637624 4091 raft_consensus.cc:348] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } }
I20260809 09:30:26.637836 4091 raft_consensus.cc:374] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:26.637893 4091 raft_consensus.cc:729] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb88197ab8e47f1979f0b882e18af33, State: Initialized, Role: FOLLOWER
I20260809 09:30:26.638285 4091 consensus_queue.cc:260] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } }
I20260809 09:30:26.638832 3838 heartbeater.cc:503] Master 127.9.102.62:40855 was elected leader, sending a full tablet report...
I20260809 09:30:26.638796 4093 raft_consensus.cc:348] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } }
I20260809 09:30:26.638796 4089 raft_consensus.cc:348] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } }
I20260809 09:30:26.639051 4089 raft_consensus.cc:374] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:26.639043 4093 raft_consensus.cc:374] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:26.639102 4093 raft_consensus.cc:729] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1737b2a81854cf0ad1384f1e7538922, State: Initialized, Role: FOLLOWER
I20260809 09:30:26.639102 4089 raft_consensus.cc:729] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82dd3517f63f4ee2aabdcc6e0716c097, State: Initialized, Role: FOLLOWER
I20260809 09:30:26.639499 4089 consensus_queue.cc:260] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } }
I20260809 09:30:26.639499 4093 consensus_queue.cc:260] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } }
I20260809 09:30:26.640230 4091 ts_tablet_manager.cc:1428] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:30:26.640377 4089 ts_tablet_manager.cc:1428] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260809 09:30:26.640377 4093 ts_tablet_manager.cc:1428] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260809 09:30:26.640663 4058 heartbeater.cc:503] Master 127.9.102.62:40855 was elected leader, sending a full tablet report...
I20260809 09:30:26.641459 3596 heartbeater.cc:503] Master 127.9.102.62:40855 was elected leader, sending a full tablet report...
W20260809 09:30:26.670179 3598 tablet.cc:2369] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:30:26.757151 4104 raft_consensus.cc:482] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:30:26.757332 4104 raft_consensus.cc:504] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } }
I20260809 09:30:26.758316 4104 leader_election.cc:288] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269), 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267)
I20260809 09:30:26.761974 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.758617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:57403 (local address 127.9.102.3:46267)
0809 09:30:26.758737 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:26.758742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:26.759014 (+ 272us) server_negotiation.cc:407] Connection header received
0809 09:30:26.759151 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:26.759154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:26.759196 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:26.759272 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:26.759652 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.760141 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.761075 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.761456 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.761571 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.761630 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:26.761794 (+ 164us) server_negotiation.cc:299] Negotiation successful
0809 09:30:26.761860 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 09:30:26.761993 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.758507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:46267 (local address 127.9.102.1:57403)
0809 09:30:26.758899 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:26.758912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:26.759049 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:26.759376 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:26.759379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:26.759395 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:26.759536 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.759542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.760240 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.760245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.760957 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.760965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.761573 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.761576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.761671 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.761697 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 09:30:26.761775 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 09:30:26.761857 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":56,"thread_start_us":86,"threads_started":1}
I20260809 09:30:26.761997 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.758620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:49575 (local address 127.9.102.2:34269)
0809 09:30:26.758737 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:26.758742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:26.758888 (+ 146us) server_negotiation.cc:407] Connection header received
0809 09:30:26.759088 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:26.759094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:26.759144 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:26.759190 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:26.759762 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.760204 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.761114 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.761480 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.761590 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.761646 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:26.761820 (+ 174us) server_negotiation.cc:299] Negotiation successful
0809 09:30:26.761887 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:30:26.762024 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.758507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34269 (local address 127.9.102.1:49575)
0809 09:30:26.758802 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:26.758813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:26.758940 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:26.759332 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:26.759334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:26.759347 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:26.759650 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.759662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.760305 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.760309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.761000 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:26.761007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.761595 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.761598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:26.761700 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.761720 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 09:30:26.761804 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 09:30:26.761894 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":182,"thread_start_us":64,"threads_started":1}
I20260809 09:30:26.762518 3752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58aefdcbb9444479e6df8d68dff1c55" candidate_uuid: "e1737b2a81854cf0ad1384f1e7538922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb88197ab8e47f1979f0b882e18af33" is_pre_election: true
I20260809 09:30:26.762651 3984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58aefdcbb9444479e6df8d68dff1c55" candidate_uuid: "e1737b2a81854cf0ad1384f1e7538922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" is_pre_election: true
I20260809 09:30:26.762766 3752 raft_consensus.cc:2454] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1737b2a81854cf0ad1384f1e7538922 in term 0.
I20260809 09:30:26.762919 3984 raft_consensus.cc:2454] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1737b2a81854cf0ad1384f1e7538922 in term 0.
I20260809 09:30:26.763234 3400 leader_election.cc:302] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9eb88197ab8e47f1979f0b882e18af33, e1737b2a81854cf0ad1384f1e7538922; no voters:
I20260809 09:30:26.763486 4104 raft_consensus.cc:2781] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:30:26.763568 4104 raft_consensus.cc:482] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:30:26.763595 4104 raft_consensus.cc:3037] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:26.764631 4104 raft_consensus.cc:504] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } }
I20260809 09:30:26.764986 4104 leader_election.cc:288] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 1 election: Requested vote from peers 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269), 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267)
I20260809 09:30:26.765295 3752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58aefdcbb9444479e6df8d68dff1c55" candidate_uuid: "e1737b2a81854cf0ad1384f1e7538922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb88197ab8e47f1979f0b882e18af33"
I20260809 09:30:26.765295 3984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58aefdcbb9444479e6df8d68dff1c55" candidate_uuid: "e1737b2a81854cf0ad1384f1e7538922" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82dd3517f63f4ee2aabdcc6e0716c097"
I20260809 09:30:26.765409 3752 raft_consensus.cc:3037] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:26.765409 3984 raft_consensus.cc:3037] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:26.766568 3752 raft_consensus.cc:2454] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1737b2a81854cf0ad1384f1e7538922 in term 1.
I20260809 09:30:26.766569 3984 raft_consensus.cc:2454] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1737b2a81854cf0ad1384f1e7538922 in term 1.
I20260809 09:30:26.766834 3399 leader_election.cc:302] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82dd3517f63f4ee2aabdcc6e0716c097, e1737b2a81854cf0ad1384f1e7538922; no voters:
I20260809 09:30:26.766969 4104 raft_consensus.cc:2781] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:30:26.767179 4104 raft_consensus.cc:686] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 1 LEADER]: Becoming Leader. State: Replica: e1737b2a81854cf0ad1384f1e7538922, State: Running, Role: LEADER
I20260809 09:30:26.767459 4104 consensus_queue.cc:237] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } }
I20260809 09:30:26.770349 3276 catalog_manager.cc:5495] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1737b2a81854cf0ad1384f1e7538922 (127.9.102.1). New cstate: current_term: 1 leader_uuid: "e1737b2a81854cf0ad1384f1e7538922" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } health_report { overall_health: UNKNOWN } } }
I20260809 09:30:26.802345 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.799517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.9.102.1:38573)
0809 09:30:26.799622 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:26.799642 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:26.799655 (+ 13us) server_negotiation.cc:407] Connection header received
0809 09:30:26.799770 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:26.799772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:26.799816 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:26.799911 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:30:26.800157 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.800599 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.801307 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:26.801485 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:26.801553 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:26.801756 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:30:26.802075 (+ 319us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:30:26.802151 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:26.802215 (+ 64us) server_negotiation.cc:299] Negotiation successful
0809 09:30:26.802254 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 09:30:26.807518 3752 raft_consensus.cc:1261] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33 [term 1 FOLLOWER]: Refusing update from remote peer e1737b2a81854cf0ad1384f1e7538922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:30:26.807518 3984 raft_consensus.cc:1261] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097 [term 1 FOLLOWER]: Refusing update from remote peer e1737b2a81854cf0ad1384f1e7538922: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:30:26.808254 4113 consensus_queue.cc:1035] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:30:26.808436 4104 consensus_queue.cc:1035] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:30:26.815685 4117 mvcc.cc:205] Tried to move back new op lower bound from 7316553018596884480 to 7316553018442207232. Current Snapshot: MvccSnapshot[applied={T|T < 7316553018596884480}]
I20260809 09:30:26.816211 4118 mvcc.cc:205] Tried to move back new op lower bound from 7316553018596884480 to 7316553018442207232. Current Snapshot: MvccSnapshot[applied={T|T < 7316553018596884480}]
I20260809 09:30:26.823793 3276 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:30:26.823915 3276 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:30:26.826126 3276 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Sending DeleteTablet for 3 replicas of tablet e58aefdcbb9444479e6df8d68dff1c55
I20260809 09:30:26.826759 3487 tablet_service.cc:1514] Processing DeleteTablet for tablet e58aefdcbb9444479e6df8d68dff1c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:26 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:51116
I20260809 09:30:26.826812 3953 tablet_service.cc:1514] Processing DeleteTablet for tablet e58aefdcbb9444479e6df8d68dff1c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:26 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:46724
I20260809 09:30:26.826818 3718 tablet_service.cc:1514] Processing DeleteTablet for tablet e58aefdcbb9444479e6df8d68dff1c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:26 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:41140
I20260809 09:30:26.827222 4134 tablet_replica.cc:330] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: stopping tablet replica
I20260809 09:30:26.827332 4132 tablet_replica.cc:330] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097: stopping tablet replica
I20260809 09:30:26.827490 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 3366
I20260809 09:30:26.827507 4134 raft_consensus.cc:2229] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:30:26.827610 4132 raft_consensus.cc:2229] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:30:26.827824 4134 raft_consensus.cc:2258] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:30:26.827852 4132 raft_consensus.cc:2258] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:30:26.828167 4136 tablet_replica.cc:330] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33: stopping tablet replica
I20260809 09:30:26.828351 4136 raft_consensus.cc:2229] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:30:26.828524 4136 raft_consensus.cc:2258] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:30:26.834434 4136 ts_tablet_manager.cc:1905] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:30:26.836452 4132 ts_tablet_manager.cc:1905] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:30:26.837796 4136 ts_tablet_manager.cc:1918] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:30:26.837898 4136 log.cc:1196] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/e58aefdcbb9444479e6df8d68dff1c55
I20260809 09:30:26.838272 4136 ts_tablet_manager.cc:1939] T e58aefdcbb9444479e6df8d68dff1c55 P 9eb88197ab8e47f1979f0b882e18af33: Deleting consensus metadata
I20260809 09:30:26.838495 4132 ts_tablet_manager.cc:1918] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:30:26.838604 4132 log.cc:1196] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/e58aefdcbb9444479e6df8d68dff1c55
I20260809 09:30:26.838933 4132 ts_tablet_manager.cc:1939] T e58aefdcbb9444479e6df8d68dff1c55 P 82dd3517f63f4ee2aabdcc6e0716c097: Deleting consensus metadata
I20260809 09:30:26.839479 3238 catalog_manager.cc:4841] TS 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269): tablet e58aefdcbb9444479e6df8d68dff1c55 (table test-table [id=5412f7a5a3434bf7b8c4c792e8417099]) successfully deleted
I20260809 09:30:26.840016 3236 catalog_manager.cc:4841] TS 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267): tablet e58aefdcbb9444479e6df8d68dff1c55 (table test-table [id=5412f7a5a3434bf7b8c4c792e8417099]) successfully deleted
W20260809 09:30:26.842044 3244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:38573 (error 108)
I20260809 09:30:26.842343 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 3609
I20260809 09:30:26.843418 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:26.843066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38573 (local address 127.0.0.1:51134)
0809 09:30:26.843211 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:26.843299 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:38573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 09:30:26.843667 3244 catalog_manager.cc:4584] TS e1737b2a81854cf0ad1384f1e7538922 (127.9.102.1:38573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e58aefdcbb9444479e6df8d68dff1c55: Network error: Client connection negotiation failed: client connection to 127.9.102.1:38573: connect: Connection refused (error 111)
I20260809 09:30:26.852816 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 3845
I20260809 09:30:26.872830 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 3203
I20260809 09:30:46.880607 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40855
--webserver_interface=127.9.102.62
--webserver_port=46207
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:40855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:30:46.982425 4143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:46.982664 4143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:46.982726 4143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:46.986155 4143 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 09:30:46.986215 4143 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 09:30:46.986239 4143 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 09:30:46.986264 4143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:30:46.986299 4143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:46.986339 4143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:46.986385 4143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:30:46.986420 4143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:30:46.990811 4143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:40855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=46207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:46.991773 4143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:46.992692 4143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:47.005004 4152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:47.005245 4150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:47.005046 4149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:47.005307 4143 server_base.cc:1034] running on GCE node
I20260809 09:30:47.005915 4143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:47.006925 4143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:47.008149 4143 hybrid_clock.cc:648] HybridClock initialized: now 1786267847008138 us; error 23 us; skew 500 ppm
Aug 09 09:30:47 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:47.010722 4143 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:30:47.011543 4143 webserver.cc:460] Webserver started at http://127.9.102.62:46207/ using document root <none> and password file <none>
I20260809 09:30:47.012053 4143 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:47.012178 4143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:47.015612 4143 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260809 09:30:47.017654 4162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:47.018733 4143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:30:47.018872 4143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "24691abac8754f36bd7a6146d48893d3"
format_stamp: "Formatted at 2026-08-09 09:30:25 on dist-test-slave-7vjv"
server_key: "17be5d4af0a956a902d4e039671fb4fc"
server_key_iv: "a706a46c360a3f5d74b748b4c5b0e0ce"
server_key_version: "encryptionkey@0"
I20260809 09:30:47.019393 4143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:47.029394 4143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:47.031714 4143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:47.031883 4143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:47.038290 4143 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:40855
I20260809 09:30:47.038580 4248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:40855 every 8 connection(s)
I20260809 09:30:47.039336 4143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:30:47.043390 4251 sys_catalog.cc:263] Verifying existing consensus state
I20260809 09:30:47.044976 4251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Bootstrap starting.
I20260809 09:30:47.047781 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 4143
I20260809 09:30:47.048241 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:38573
--local_ip_for_outbound_sockets=127.9.102.1
--tserver_master_addrs=127.9.102.62:40855
--webserver_port=42599
--webserver_interface=127.9.102.1
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:30:47.060482 4251 log.cc:824] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:47.068055 4251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 09:30:47.068428 4251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Bootstrap complete.
I20260809 09:30:47.071928 4251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } }
I20260809 09:30:47.072341 4251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24691abac8754f36bd7a6146d48893d3, State: Initialized, Role: FOLLOWER
I20260809 09:30:47.072829 4251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } }
I20260809 09:30:47.072974 4251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:30:47.073071 4251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:30:47.073156 4251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 1 FOLLOWER]: Advancing to term 2
I20260809 09:30:47.074857 4251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } }
I20260809 09:30:47.075171 4251 leader_election.cc:302] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24691abac8754f36bd7a6146d48893d3; no voters:
I20260809 09:30:47.075466 4251 leader_election.cc:288] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 09:30:47.075531 4261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 09:30:47.075759 4261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 2 LEADER]: Becoming Leader. State: Replica: 24691abac8754f36bd7a6146d48893d3, State: Running, Role: LEADER
I20260809 09:30:47.076074 4261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } }
I20260809 09:30:47.076575 4251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:30:47.076946 4261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24691abac8754f36bd7a6146d48893d3. Latest consensus state: current_term: 2 leader_uuid: "24691abac8754f36bd7a6146d48893d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } } }
I20260809 09:30:47.077132 4261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:47.077272 4262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "24691abac8754f36bd7a6146d48893d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } } }
I20260809 09:30:47.077445 4262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: This master's current role is: LEADER
I20260809 09:30:47.077527 4268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:30:47.081871 4268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:30:47.082535 4268 catalog_manager.cc:1259] Loaded cluster ID: 61aead23013f41c9b608f3969d91ece0
I20260809 09:30:47.082597 4268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:30:47.084941 4268 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:30:47.085618 4268 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Loaded TSK: 0
I20260809 09:30:47.113425 4268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Generated new TSK 1
I20260809 09:30:47.113664 4268 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 09:30:47.153520 4256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:47.153800 4256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:47.153878 4256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:47.157785 4256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:47.157858 4256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:47.157958 4256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:30:47.162829 4256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:38573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=42599
--tserver_master_addrs=127.9.102.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:47.163636 4256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:47.164572 4256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:47.174654 4300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:47.174669 4299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:47.174666 4303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:47.175277 4256 server_base.cc:1034] running on GCE node
I20260809 09:30:47.175643 4256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:47.176280 4256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:47.177467 4256 hybrid_clock.cc:648] HybridClock initialized: now 1786267847177432 us; error 36 us; skew 500 ppm
Aug 09 09:30:47 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:47.180325 4256 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:30:47.181281 4256 webserver.cc:460] Webserver started at http://127.9.102.1:42599/ using document root <none> and password file <none>
I20260809 09:30:47.181859 4256 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:47.181957 4256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:47.186383 4256 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 09:30:47.188658 4321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:47.190112 4256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:30:47.190227 4256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "e1737b2a81854cf0ad1384f1e7538922"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "24eef352574eab1027b6b62646d1a9d9"
server_key_iv: "9d7c5b318c1f8738cfbc7fb93bbc3d07"
server_key_version: "encryptionkey@0"
I20260809 09:30:47.190837 4256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:47.210238 4256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:47.213496 4256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:47.213661 4256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:47.214270 4256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:47.215435 4332 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 09:30:47.219135 4256 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 09:30:47.219225 4256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260809 09:30:47.219301 4256 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 09:30:47.221342 4256 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 09:30:47.221411 4256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260809 09:30:47.221716 4332 tablet_bootstrap.cc:492] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Bootstrap starting.
I20260809 09:30:47.230911 4256 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:38573
I20260809 09:30:47.231205 4514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:38573 every 8 connection(s)
I20260809 09:30:47.231982 4256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:30:47.234943 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 4256
I20260809 09:30:47.235453 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:34269
--local_ip_for_outbound_sockets=127.9.102.2
--tserver_master_addrs=127.9.102.62:40855
--webserver_port=32933
--webserver_interface=127.9.102.2
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:30:47.241010 4332 log.cc:824] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Log is configured to *not* fsync() on all Append() calls
Aug 09 09:30:47 dist-test-slave-7vjv krb5kdc[3187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:47.248900 4332 tablet_bootstrap.cc:492] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 09:30:47.249359 4332 tablet_bootstrap.cc:492] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Bootstrap complete.
I20260809 09:30:47.250663 4332 ts_tablet_manager.cc:1397] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.008s
I20260809 09:30:47.250893 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.233979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:60649)
0809 09:30:47.234558 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:47.234625 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:47.235708 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:47.237045 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:47.237053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:47.237646 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:47.238280 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.238299 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.239693 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.239697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.240969 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.240983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.242136 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.242140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.242325 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.243033 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:47.243065 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:47.245042 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:47.247577 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:47.247586 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:47.247600 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:47.247971 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:47.248372 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:47.248377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:47.248381 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:47.248583 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:47.249245 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:47.249251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:47.249533 (+ 282us) client_negotiation.cc:769] Sending connection context
0809 09:30:47.249742 (+ 209us) client_negotiation.cc:240] Negotiation successful
0809 09:30:47.250034 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":159,"threads_started":1}
I20260809 09:30:47.251011 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.233982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:60649 (local address 127.9.102.62:40855)
0809 09:30:47.234341 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:47.234360 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:47.234885 (+ 525us) server_negotiation.cc:407] Connection header received
0809 09:30:47.236194 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:47.236213 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:47.236595 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:47.236941 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:47.238485 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.239529 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.241126 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.241960 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.242156 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.245257 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:47.245264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:47.245279 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:47.245309 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:47.247429 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:47.248116 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:47.248121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:47.248126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:47.248265 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:47.248699 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:47.248704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:47.248707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:47.249120 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:47.249261 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:47.249877 (+ 616us) server_negotiation.cc:299] Negotiation successful
0809 09:30:47.250170 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":109,"threads_started":1}
I20260809 09:30:47.253561 4515 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40855
I20260809 09:30:47.253221 4332 raft_consensus.cc:348] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } }
I20260809 09:30:47.253721 4332 raft_consensus.cc:729] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1737b2a81854cf0ad1384f1e7538922, State: Initialized, Role: FOLLOWER
I20260809 09:30:47.253780 4515 heartbeater.cc:464] Registering TS with master...
I20260809 09:30:47.254175 4332 consensus_queue.cc:260] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } }
I20260809 09:30:47.254259 4515 heartbeater.cc:511] Master 127.9.102.62:40855 requested a full tablet report, sending...
I20260809 09:30:47.255062 4332 ts_tablet_manager.cc:1428] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260809 09:30:47.256821 4191 ts_manager.cc:194] Registered new tserver with Master: e1737b2a81854cf0ad1384f1e7538922 (127.9.102.1:38573)
I20260809 09:30:47.260391 4191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:60649
I20260809 09:30:47.262903 4515 heartbeater.cc:503] Master 127.9.102.62:40855 was elected leader, sending a full tablet report...
Aug 09 09:30:47 dist-test-slave-7vjv krb5kdc[3187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.1@KRBTEST.COM
I20260809 09:30:47.269500 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.259755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38573 (local address 127.0.0.1:42826)
0809 09:30:47.260364 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:47.260394 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:47.260534 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:47.260938 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:47.260943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:47.261328 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:47.261456 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.261464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.262724 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.262728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.263583 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.263594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.264287 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.264290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.264396 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.264825 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:47.264835 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:47.266173 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:47.267778 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:47.267788 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:47.267793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:47.267998 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:47.268350 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:47.268353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:47.268354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:47.268439 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:47.268869 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:47.268875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:47.268996 (+ 121us) client_negotiation.cc:769] Sending connection context
0809 09:30:47.269159 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 09:30:47.269338 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":89,"threads_started":1}
I20260809 09:30:47.269551 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.259807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42826 (local address 127.9.102.1:38573)
0809 09:30:47.260151 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:47.260161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:47.260480 (+ 319us) server_negotiation.cc:407] Connection header received
0809 09:30:47.260669 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:47.260678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:47.260806 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:47.260933 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:47.261584 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.262594 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.263709 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.264200 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.264292 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.266304 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:47.266318 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:47.266326 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:47.266344 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:47.267673 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:47.268151 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:47.268158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:47.268163 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:47.268248 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:47.268542 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:47.268545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:47.268547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:47.268753 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:47.268888 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:47.269249 (+ 361us) server_negotiation.cc:299] Negotiation successful
0809 09:30:47.269397 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":246,"thread_start_us":87,"threads_started":1}
I20260809 09:30:47.270392 4395 tablet_service.cc:1514] Processing DeleteTablet for tablet e58aefdcbb9444479e6df8d68dff1c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:26 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:42826
I20260809 09:30:47.271155 4530 tablet_replica.cc:330] stopping tablet replica
I20260809 09:30:47.271432 4530 raft_consensus.cc:2229] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:30:47.271668 4530 raft_consensus.cc:2258] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:30:47.272624 4530 ts_tablet_manager.cc:1905] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:30:47.276710 4530 ts_tablet_manager.cc:1918] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:30:47.276809 4530 log.cc:1196] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal/wals/e58aefdcbb9444479e6df8d68dff1c55
I20260809 09:30:47.277070 4530 ts_tablet_manager.cc:1939] T e58aefdcbb9444479e6df8d68dff1c55 P e1737b2a81854cf0ad1384f1e7538922: Deleting consensus metadata
I20260809 09:30:47.278800 4167 catalog_manager.cc:4841] TS e1737b2a81854cf0ad1384f1e7538922 (127.9.102.1:38573): tablet e58aefdcbb9444479e6df8d68dff1c55 (table test-table [id=5412f7a5a3434bf7b8c4c792e8417099]) successfully deleted
W20260809 09:30:47.347549 4521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:47.347747 4521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:47.347812 4521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:47.351285 4521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:47.351329 4521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:47.351393 4521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:30:47.355775 4521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:34269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=32933
--tserver_master_addrs=127.9.102.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:47.356637 4521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:47.357456 4521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:47.365410 4538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:47.365427 4557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:47.365427 4554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:47.365605 4521 server_base.cc:1034] running on GCE node
I20260809 09:30:47.365935 4521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:47.366485 4521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:47.367671 4521 hybrid_clock.cc:648] HybridClock initialized: now 1786267847367660 us; error 21 us; skew 500 ppm
Aug 09 09:30:47 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:47.370105 4521 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:30:47.370810 4521 webserver.cc:460] Webserver started at http://127.9.102.2:32933/ using document root <none> and password file <none>
I20260809 09:30:47.371248 4521 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:47.371311 4521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:47.374553 4521 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:30:47.376242 4570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:47.377341 4521 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:47.377434 4521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "9eb88197ab8e47f1979f0b882e18af33"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "190b53b1c8fc7622a603534948d7cdb6"
server_key_iv: "6727519ec2395abd46ce8d04881c6aa1"
server_key_version: "encryptionkey@0"
I20260809 09:30:47.377875 4521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:47.386917 4521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:47.389544 4521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:47.389678 4521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:47.390136 4521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:47.390833 4521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:30:47.390877 4521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:47.390908 4521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:30:47.390934 4521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:47.399770 4521 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:34269
I20260809 09:30:47.399828 4775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:34269 every 8 connection(s)
I20260809 09:30:47.400681 4521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:30:47.403635 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 4521
I20260809 09:30:47.404055 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:46267
--local_ip_for_outbound_sockets=127.9.102.3
--tserver_master_addrs=127.9.102.62:40855
--webserver_port=33889
--webserver_interface=127.9.102.3
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 09:30:47 dist-test-slave-7vjv krb5kdc[3187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:47.413714 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.402588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:58653 (local address 127.9.102.62:40855)
0809 09:30:47.402708 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:47.402712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:47.403271 (+ 559us) server_negotiation.cc:407] Connection header received
0809 09:30:47.404624 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:47.404631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:47.404703 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:47.404809 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:47.406205 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.406725 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.407780 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.408003 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.408139 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.410381 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:47.410389 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:47.410391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:47.410415 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:47.411757 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:47.412291 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:47.412295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:47.412297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:47.412386 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:47.412727 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:47.412730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:47.412731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:47.412896 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:47.412998 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:47.413431 (+ 433us) server_negotiation.cc:299] Negotiation successful
0809 09:30:47.413555 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 09:30:47.414371 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.402711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:58653)
0809 09:30:47.403121 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:47.403157 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:47.404362 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:47.404998 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:47.405008 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:47.405457 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:47.406039 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.406055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.406854 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.406858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.407646 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.407656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.408163 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.408167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.408331 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.408895 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:47.408911 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:47.410227 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:47.411870 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:47.411876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:47.411887 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:47.412187 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:47.412485 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:47.412487 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:47.412489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:47.412627 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:47.412999 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:47.413006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:47.413215 (+ 209us) client_negotiation.cc:769] Sending connection context
0809 09:30:47.413376 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 09:30:47.413655 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260809 09:30:47.415552 4778 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40855
I20260809 09:30:47.415757 4778 heartbeater.cc:464] Registering TS with master...
I20260809 09:30:47.416217 4778 heartbeater.cc:511] Master 127.9.102.62:40855 requested a full tablet report, sending...
I20260809 09:30:47.417281 4191 ts_manager.cc:194] Registered new tserver with Master: 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269)
I20260809 09:30:47.417959 4191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:58653
W20260809 09:30:47.504541 4786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:30:47.504736 4786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:30:47.504772 4786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:30:47.508411 4786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:30:47.508455 4786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:30:47.508504 4786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:30:47.512921 4786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:46267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=33889
--tserver_master_addrs=127.9.102.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:30:47.513633 4786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:30:47.514389 4786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:30:47.522449 4799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:30:47.522509 4797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:47.522629 4786 server_base.cc:1034] running on GCE node
W20260809 09:30:47.522787 4802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:30:47.523182 4786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:30:47.523693 4786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:30:47.524873 4786 hybrid_clock.cc:648] HybridClock initialized: now 1786267847524855 us; error 25 us; skew 500 ppm
Aug 09 09:30:47 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:30:47.531985 4786 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:30:47.532810 4786 webserver.cc:460] Webserver started at http://127.9.102.3:33889/ using document root <none> and password file <none>
I20260809 09:30:47.533300 4786 fs_manager.cc:359] Metadata directory not provided
I20260809 09:30:47.533406 4786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:30:47.536739 4786 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:30:47.538690 4814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:47.539803 4786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:30:47.539938 4786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "82dd3517f63f4ee2aabdcc6e0716c097"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "bd3c7a4fab525e914c9e8e4661c6cd7a"
server_key_iv: "90d1c3fa1843073a8d575c31f63fdf1c"
server_key_version: "encryptionkey@0"
I20260809 09:30:47.540508 4786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:30:47.551702 4786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:30:47.554637 4786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:30:47.554813 4786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:30:47.555341 4786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:30:47.556115 4786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:30:47.556190 4786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:47.556268 4786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:30:47.556324 4786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:30:47.565114 4786 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:46267
I20260809 09:30:47.565137 5023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:46267 every 8 connection(s)
I20260809 09:30:47.565932 4786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:30:47.571192 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 4786
Aug 09 09:30:47 dist-test-slave-7vjv krb5kdc[3187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267847, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:30:47.579355 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.567406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:41233 (local address 127.9.102.62:40855)
0809 09:30:47.567531 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:47.567535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:47.568690 (+ 1155us) server_negotiation.cc:407] Connection header received
0809 09:30:47.569452 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:47.569455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:47.569511 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:47.569607 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:47.570580 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.571122 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.572436 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.572801 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.572931 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.575509 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:47.575512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:47.575514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:47.575541 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:47.577093 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:47.577674 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:47.577678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:47.577680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:47.577821 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:47.578241 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:47.578243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:47.578244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:47.578406 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:47.578546 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:47.579069 (+ 523us) server_negotiation.cc:299] Negotiation successful
0809 09:30:47.579185 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:30:47.580014 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.567679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:41233)
0809 09:30:47.568560 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:47.568595 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:47.569268 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:47.569759 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:47.569764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:47.570061 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:30:47.570438 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.570449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.571251 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.571256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.572295 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.572324 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.572943 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.572946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.573070 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.573582 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:30:47.573600 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:30:47.575362 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:30:47.577215 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:47.577220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:47.577233 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:47.577571 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:47.577922 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:30:47.577924 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:30:47.577925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:30:47.578136 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:30:47.578560 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:30:47.578568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:30:47.578836 (+ 268us) client_negotiation.cc:769] Sending connection context
0809 09:30:47.578973 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 09:30:47.579253 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":584,"threads_started":1}
I20260809 09:30:47.580726 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.571834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.9.102.62:40855)
0809 09:30:47.572074 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:47.572078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:47.572250 (+ 172us) server_negotiation.cc:407] Connection header received
0809 09:30:47.572358 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:47.572363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:47.572414 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:47.572489 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:47.573411 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.574138 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.575411 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.575609 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.575715 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.576970 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:47.576974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:47.576976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:47.577004 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:47.578682 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:47.579106 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:47.579109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:47.579111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:47.579197 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:47.579521 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:47.579526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:47.579528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:47.579722 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:47.579937 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:47.580070 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 09:30:47.580524 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":80,"threads_started":1}
I20260809 09:30:47.581413 5025 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40855
I20260809 09:30:47.581629 5025 heartbeater.cc:464] Registering TS with master...
I20260809 09:30:47.582094 5025 heartbeater.cc:511] Master 127.9.102.62:40855 requested a full tablet report, sending...
I20260809 09:30:47.582984 4191 ts_manager.cc:194] Registered new tserver with Master: 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267)
I20260809 09:30:47.583516 4191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:41233
I20260809 09:30:47.593820 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:30:47.597137 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.594383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58322 (local address 127.9.102.62:40855)
0809 09:30:47.594474 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:47.594477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:47.594639 (+ 162us) server_negotiation.cc:407] Connection header received
0809 09:30:47.594745 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:47.594747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:47.594816 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:47.594884 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:30:47.595048 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.595430 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.596235 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.596429 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.596497 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.596782 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:30:47.596895 (+ 113us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 09:30:47.597062 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58322: token expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260809 09:30:47.597280 5034 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58322: token expired
W20260809 09:30:47.597538 4062 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.102.62:40855)
I20260809 09:30:47.603504 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.597919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.9.102.62:40855)
0809 09:30:47.598009 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:47.598011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:47.598021 (+ 10us) server_negotiation.cc:407] Connection header received
0809 09:30:47.598090 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:47.598092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:47.598118 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:47.598173 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:30:47.598646 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.599108 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.599761 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.599929 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.599992 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.601043 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:30:47.601045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:30:47.601047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:30:47.601066 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:30:47.602362 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:47.602738 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:47.602740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:47.602741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:47.602801 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:30:47.603012 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:30:47.603014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:30:47.603015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:30:47.603139 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:30:47.603222 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:47.603359 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 09:30:47.603426 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260809 09:30:47.605479 4191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:30:47.608361 4191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:30:47.614634 4395 tablet_service.cc:1467] Processing CreateTablet for tablet dd22c7388f1f4a71b8b8e6ece181b30d (DEFAULT_TABLE table=test-table [id=41b6400123cc4b60b3f500145ed2308e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:30:47.615283 4395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd22c7388f1f4a71b8b8e6ece181b30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:47.617893 4332 tablet_bootstrap.cc:492] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: Bootstrap starting.
I20260809 09:30:47.618193 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.613724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:46267 (local address 127.0.0.1:42706)
0809 09:30:47.613821 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:47.613833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:47.613889 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:47.614593 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:47.614595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:47.614611 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:47.614705 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.614710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.616058 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.616063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.616901 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.616911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.617811 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.617817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.617903 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.617953 (+ 50us) client_negotiation.cc:769] Sending connection context
0809 09:30:47.618019 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 09:30:47.618096 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":52}
I20260809 09:30:47.618436 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.613871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.9.102.3:46267)
0809 09:30:47.614197 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:47.614207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:47.614225 (+ 18us) server_negotiation.cc:407] Connection header received
0809 09:30:47.614283 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:47.614288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:47.614445 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:47.614852 (+ 407us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:47.615000 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.615940 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.617045 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.617664 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.617782 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.617902 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:47.618196 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 09:30:47.618298 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":79,"threads_started":1}
I20260809 09:30:47.618726 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.613959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34269 (local address 127.0.0.1:37012)
0809 09:30:47.614117 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:47.614130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:47.614204 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:47.615037 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:47.615040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:47.615057 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:47.615138 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.615144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.616162 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.616165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.617177 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.617187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.618348 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.618352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.618436 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.618466 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 09:30:47.618538 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 09:30:47.618605 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260809 09:30:47.618793 4332 tablet_bootstrap.cc:654] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:47.619349 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.614304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37012 (local address 127.9.102.2:34269)
0809 09:30:47.614583 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:47.614593 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:47.614612 (+ 19us) server_negotiation.cc:407] Connection header received
0809 09:30:47.614699 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:47.614704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:47.614903 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:47.615051 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:47.615219 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.615973 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.617302 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.618222 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.618995 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.619093 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:47.619132 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 09:30:47.619192 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":117,"threads_started":1}
I20260809 09:30:47.620332 4332 tablet_bootstrap.cc:492] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: No bootstrap required, opened a new log
I20260809 09:30:47.620427 4332 ts_tablet_manager.cc:1397] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260809 09:30:47.620777 4332 raft_consensus.cc:348] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:30:47.620893 4332 raft_consensus.cc:374] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:47.620934 4332 raft_consensus.cc:729] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1737b2a81854cf0ad1384f1e7538922, State: Initialized, Role: FOLLOWER
I20260809 09:30:47.621039 4332 consensus_queue.cc:260] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:30:47.621335 4332 ts_tablet_manager.cc:1428] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260809 09:30:47.621304 4902 tablet_service.cc:1467] Processing CreateTablet for tablet dd22c7388f1f4a71b8b8e6ece181b30d (DEFAULT_TABLE table=test-table [id=41b6400123cc4b60b3f500145ed2308e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:30:47.622339 4902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd22c7388f1f4a71b8b8e6ece181b30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:47.621963 4656 tablet_service.cc:1467] Processing CreateTablet for tablet dd22c7388f1f4a71b8b8e6ece181b30d (DEFAULT_TABLE table=test-table [id=41b6400123cc4b60b3f500145ed2308e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:30:47.622893 4656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd22c7388f1f4a71b8b8e6ece181b30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:30:47.629433 5042 tablet_bootstrap.cc:492] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33: Bootstrap starting.
I20260809 09:30:47.629433 5043 tablet_bootstrap.cc:492] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097: Bootstrap starting.
I20260809 09:30:47.631651 5043 tablet_bootstrap.cc:654] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:47.632194 5042 tablet_bootstrap.cc:654] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33: Neither blocks nor log segments found. Creating new log.
I20260809 09:30:47.632870 5043 log.cc:824] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:47.632870 5042 log.cc:824] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33: Log is configured to *not* fsync() on all Append() calls
I20260809 09:30:47.634725 5042 tablet_bootstrap.cc:492] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33: No bootstrap required, opened a new log
I20260809 09:30:47.634727 5043 tablet_bootstrap.cc:492] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097: No bootstrap required, opened a new log
I20260809 09:30:47.634907 5042 ts_tablet_manager.cc:1397] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 09:30:47.634907 5043 ts_tablet_manager.cc:1397] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260809 09:30:47.638603 5042 raft_consensus.cc:348] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:30:47.638603 5043 raft_consensus.cc:348] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:30:47.638864 5042 raft_consensus.cc:374] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:47.638864 5043 raft_consensus.cc:374] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:30:47.638917 5043 raft_consensus.cc:729] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82dd3517f63f4ee2aabdcc6e0716c097, State: Initialized, Role: FOLLOWER
I20260809 09:30:47.638917 5042 raft_consensus.cc:729] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eb88197ab8e47f1979f0b882e18af33, State: Initialized, Role: FOLLOWER
I20260809 09:30:47.639321 5043 consensus_queue.cc:260] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:30:47.639321 5042 consensus_queue.cc:260] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:30:47.639895 5025 heartbeater.cc:503] Master 127.9.102.62:40855 was elected leader, sending a full tablet report...
I20260809 09:30:47.640142 4778 heartbeater.cc:503] Master 127.9.102.62:40855 was elected leader, sending a full tablet report...
I20260809 09:30:47.640259 5043 ts_tablet_manager.cc:1428] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260809 09:30:47.640259 5042 ts_tablet_manager.cc:1428] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260809 09:30:47.651546 4779 tablet.cc:2369] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:30:47.710791 5051 raft_consensus.cc:482] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:30:47.710963 5051 raft_consensus.cc:504] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:30:47.712018 5051 leader_election.cc:288] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269), e1737b2a81854cf0ad1384f1e7538922 (127.9.102.1:38573)
I20260809 09:30:47.715763 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.712220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34269 (local address 127.9.102.3:43463)
0809 09:30:47.712395 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:47.712409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:47.712490 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:47.712772 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:47.712777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:47.712797 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:47.712928 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.712938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.713685 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.713689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.714674 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.714683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.715336 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.715339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.715437 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.715470 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 09:30:47.715555 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 09:30:47.715620 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260809 09:30:47.715777 5040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.712281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:43463 (local address 127.9.102.2:34269)
0809 09:30:47.712420 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:47.712425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:47.712444 (+ 19us) server_negotiation.cc:407] Connection header received
0809 09:30:47.712606 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:47.712609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:47.712668 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:47.712744 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:47.713032 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.713584 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.714787 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.715228 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.715303 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.715348 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:47.715578 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 09:30:47.715660 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:30:47.715991 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.712266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38573 (local address 127.9.102.3:37895)
0809 09:30:47.712610 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:47.712623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:30:47.712725 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:30:47.713011 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:30:47.713015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:30:47.713039 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:30:47.713141 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.713147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.713916 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.713922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.714855 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:30:47.714868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.715573 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.715576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:30:47.715689 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.715711 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 09:30:47.715791 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 09:30:47.715862 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":111,"threads_started":1}
I20260809 09:30:47.716025 4529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.712413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:37895 (local address 127.9.102.1:38573)
0809 09:30:47.712521 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:47.712528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:47.712701 (+ 173us) server_negotiation.cc:407] Connection header received
0809 09:30:47.712805 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:47.712808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:47.712874 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:47.712976 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:30:47.713245 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.713792 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.714973 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.715431 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.715539 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.715689 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:47.715817 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 09:30:47.715894 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:30:47.716550 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd22c7388f1f4a71b8b8e6ece181b30d" candidate_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1737b2a81854cf0ad1384f1e7538922" is_pre_election: true
I20260809 09:30:47.716699 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd22c7388f1f4a71b8b8e6ece181b30d" candidate_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb88197ab8e47f1979f0b882e18af33" is_pre_election: true
I20260809 09:30:47.716816 4440 raft_consensus.cc:2454] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82dd3517f63f4ee2aabdcc6e0716c097 in term 0.
I20260809 09:30:47.716964 4695 raft_consensus.cc:2454] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82dd3517f63f4ee2aabdcc6e0716c097 in term 0.
I20260809 09:30:47.717283 4823 leader_election.cc:302] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82dd3517f63f4ee2aabdcc6e0716c097, e1737b2a81854cf0ad1384f1e7538922; no voters:
I20260809 09:30:47.717516 5051 raft_consensus.cc:2781] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:30:47.717602 5051 raft_consensus.cc:482] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:30:47.717643 5051 raft_consensus.cc:3037] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:47.718596 5051 raft_consensus.cc:504] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:30:47.718889 5051 leader_election.cc:288] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [CANDIDATE]: Term 1 election: Requested vote from peers 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269), e1737b2a81854cf0ad1384f1e7538922 (127.9.102.1:38573)
I20260809 09:30:47.719154 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd22c7388f1f4a71b8b8e6ece181b30d" candidate_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1737b2a81854cf0ad1384f1e7538922"
I20260809 09:30:47.719184 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd22c7388f1f4a71b8b8e6ece181b30d" candidate_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eb88197ab8e47f1979f0b882e18af33"
I20260809 09:30:47.719286 4695 raft_consensus.cc:3037] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:47.719291 4440 raft_consensus.cc:3037] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:30:47.720288 4695 raft_consensus.cc:2454] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82dd3517f63f4ee2aabdcc6e0716c097 in term 1.
I20260809 09:30:47.720377 4440 raft_consensus.cc:2454] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82dd3517f63f4ee2aabdcc6e0716c097 in term 1.
I20260809 09:30:47.720583 4821 leader_election.cc:302] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82dd3517f63f4ee2aabdcc6e0716c097, 9eb88197ab8e47f1979f0b882e18af33; no voters:
I20260809 09:30:47.720728 5051 raft_consensus.cc:2781] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:30:47.720914 5051 raft_consensus.cc:686] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 1 LEADER]: Becoming Leader. State: Replica: 82dd3517f63f4ee2aabdcc6e0716c097, State: Running, Role: LEADER
I20260809 09:30:47.721230 5051 consensus_queue.cc:237] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:30:47.723971 4191 catalog_manager.cc:5495] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3). New cstate: current_term: 1 leader_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } health_report { overall_health: UNKNOWN } } }
W20260809 09:30:47.733217 4517 tablet.cc:2369] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:30:47.744742 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.741601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.9.102.3:46267)
0809 09:30:47.741720 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0809 09:30:47.741726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:30:47.741779 (+ 53us) server_negotiation.cc:407] Connection header received
0809 09:30:47.741944 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:30:47.741947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:30:47.742002 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:30:47.742119 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:30:47.742348 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.742782 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.743493 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:30:47.743681 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:30:47.743775 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:30:47.744040 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:30:47.744366 (+ 326us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:30:47.744480 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 09:30:47.744557 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 09:30:47.744603 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 09:30:47.749558 4695 raft_consensus.cc:1261] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33 [term 1 FOLLOWER]: Refusing update from remote peer 82dd3517f63f4ee2aabdcc6e0716c097: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:30:47.749557 4440 raft_consensus.cc:1261] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Refusing update from remote peer 82dd3517f63f4ee2aabdcc6e0716c097: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:30:47.750232 5054 consensus_queue.cc:1035] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:30:47.750419 5051 consensus_queue.cc:1035] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:30:47.754698 5058 mvcc.cc:205] Tried to move back new op lower bound from 7316553104376123392 to 7316553104268754944. Current Snapshot: MvccSnapshot[applied={T|T < 7316553104376123392}]
I20260809 09:30:47.760594 5066 mvcc.cc:205] Tried to move back new op lower bound from 7316553104376123392 to 7316553104268754944. Current Snapshot: MvccSnapshot[applied={T|T < 7316553104376123392}]
I20260809 09:30:47.762665 4190 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:30:47.762830 4190 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:30:47.764922 4190 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Sending DeleteTablet for 3 replicas of tablet dd22c7388f1f4a71b8b8e6ece181b30d
I20260809 09:30:47.765480 4902 tablet_service.cc:1514] Processing DeleteTablet for tablet dd22c7388f1f4a71b8b8e6ece181b30d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:47 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:42706
I20260809 09:30:47.765552 4395 tablet_service.cc:1514] Processing DeleteTablet for tablet dd22c7388f1f4a71b8b8e6ece181b30d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:47 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:42826
I20260809 09:30:47.765697 4530 tablet_replica.cc:330] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: stopping tablet replica
I20260809 09:30:47.765810 4530 raft_consensus.cc:2229] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:30:47.765867 5080 tablet_replica.cc:330] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097: stopping tablet replica
I20260809 09:30:47.765859 4656 tablet_service.cc:1514] Processing DeleteTablet for tablet dd22c7388f1f4a71b8b8e6ece181b30d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:30:47 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:37012
I20260809 09:30:47.765920 4530 raft_consensus.cc:2258] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:30:47.766145 5080 raft_consensus.cc:2229] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:30:47.766260 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 4256
I20260809 09:30:47.766474 5080 raft_consensus.cc:2258] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:30:47.766621 4530 ts_tablet_manager.cc:1905] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:30:47.775224 5081 tablet_replica.cc:330] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33: stopping tablet replica
I20260809 09:30:47.775530 5081 raft_consensus.cc:2229] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:30:47.775831 5081 raft_consensus.cc:2258] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:30:47.779762 5081 ts_tablet_manager.cc:1905] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:30:47.782588 5081 ts_tablet_manager.cc:1918] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:30:47.782663 5081 log.cc:1196] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/dd22c7388f1f4a71b8b8e6ece181b30d
I20260809 09:30:47.782930 5081 ts_tablet_manager.cc:1939] T dd22c7388f1f4a71b8b8e6ece181b30d P 9eb88197ab8e47f1979f0b882e18af33: Deleting consensus metadata
I20260809 09:30:47.790948 4166 catalog_manager.cc:4841] TS 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269): tablet dd22c7388f1f4a71b8b8e6ece181b30d (table test-table [id=41b6400123cc4b60b3f500145ed2308e]) successfully deleted
I20260809 09:30:47.791862 5080 ts_tablet_manager.cc:1905] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:30:47.795048 5080 ts_tablet_manager.cc:1918] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:30:47.795154 5080 log.cc:1196] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/dd22c7388f1f4a71b8b8e6ece181b30d
I20260809 09:30:47.795557 5080 ts_tablet_manager.cc:1939] T dd22c7388f1f4a71b8b8e6ece181b30d P 82dd3517f63f4ee2aabdcc6e0716c097: Deleting consensus metadata
I20260809 09:30:47.796675 4164 catalog_manager.cc:4841] TS 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267): tablet dd22c7388f1f4a71b8b8e6ece181b30d (table test-table [id=41b6400123cc4b60b3f500145ed2308e]) successfully deleted
W20260809 09:30:47.797576 4167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:38573 (error 108)
I20260809 09:30:47.798152 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 4521
I20260809 09:30:47.804697 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:30:47.804384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38573 (local address 127.0.0.1:42832)
0809 09:30:47.804556 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 09:30:47.804618 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:38573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260809 09:30:47.805459 4167 catalog_manager.cc:4584] TS e1737b2a81854cf0ad1384f1e7538922 (127.9.102.1:38573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd22c7388f1f4a71b8b8e6ece181b30d: Network error: Client connection negotiation failed: client connection to 127.9.102.1:38573: connect: Connection refused (error 111)
I20260809 09:30:47.811501 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 4786
I20260809 09:30:47.825547 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 4143
I20260809 09:31:07.837692 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40855
--webserver_interface=127.9.102.62
--webserver_port=46207
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:40855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:07.938560 5084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:07.938805 5084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:07.938897 5084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:07.942662 5084 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 09:31:07.942710 5084 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 09:31:07.942728 5084 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 09:31:07.942785 5084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:07.942821 5084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:07.942867 5084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:07.942934 5084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:07.942981 5084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:07.947717 5084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:40855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=46207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:07.948616 5084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:07.949715 5084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:07.957522 5100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:07.957541 5096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:07.957541 5097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:07.957590 5084 server_base.cc:1034] running on GCE node
I20260809 09:31:07.958308 5084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:07.959266 5084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:07.960421 5084 hybrid_clock.cc:648] HybridClock initialized: now 1786267867960410 us; error 35 us; skew 500 ppm
Aug 09 09:31:07 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267867, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:07.970484 5084 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:31:07.971350 5084 webserver.cc:460] Webserver started at http://127.9.102.62:46207/ using document root <none> and password file <none>
I20260809 09:31:07.971840 5084 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:07.971899 5084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:07.975227 5084 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:07.977129 5114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:07.978199 5084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:31:07.978335 5084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "24691abac8754f36bd7a6146d48893d3"
format_stamp: "Formatted at 2026-08-09 09:30:25 on dist-test-slave-7vjv"
server_key: "17be5d4af0a956a902d4e039671fb4fc"
server_key_iv: "a706a46c360a3f5d74b748b4c5b0e0ce"
server_key_version: "encryptionkey@0"
I20260809 09:31:07.978804 5084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:08.010072 5084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:08.012698 5084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:08.012887 5084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:08.019501 5084 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:40855
I20260809 09:31:08.019761 5191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:40855 every 8 connection(s)
I20260809 09:31:08.020566 5084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:08.024362 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 5084
I20260809 09:31:08.024606 5192 sys_catalog.cc:263] Verifying existing consensus state
I20260809 09:31:08.024849 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:38573
--local_ip_for_outbound_sockets=127.9.102.1
--tserver_master_addrs=127.9.102.62:40855
--webserver_port=42599
--webserver_interface=127.9.102.1
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:31:08.026813 5192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Bootstrap starting.
I20260809 09:31:08.042583 5192 log.cc:824] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:08.052304 5192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 09:31:08.052635 5192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Bootstrap complete.
I20260809 09:31:08.055975 5192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } }
I20260809 09:31:08.056377 5192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24691abac8754f36bd7a6146d48893d3, State: Initialized, Role: FOLLOWER
I20260809 09:31:08.056918 5192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } }
I20260809 09:31:08.057044 5192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:08.057142 5192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:08.057260 5192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 2 FOLLOWER]: Advancing to term 3
I20260809 09:31:08.058885 5192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } }
I20260809 09:31:08.059180 5192 leader_election.cc:302] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24691abac8754f36bd7a6146d48893d3; no voters:
I20260809 09:31:08.059428 5192 leader_election.cc:288] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 09:31:08.059535 5202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 09:31:08.059830 5202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [term 3 LEADER]: Becoming Leader. State: Replica: 24691abac8754f36bd7a6146d48893d3, State: Running, Role: LEADER
I20260809 09:31:08.060209 5202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } }
I20260809 09:31:08.060523 5192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:08.061579 5204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24691abac8754f36bd7a6146d48893d3. Latest consensus state: current_term: 3 leader_uuid: "24691abac8754f36bd7a6146d48893d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } } }
I20260809 09:31:08.061587 5203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "24691abac8754f36bd7a6146d48893d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24691abac8754f36bd7a6146d48893d3" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40855 } } }
I20260809 09:31:08.061691 5204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:08.061699 5203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:08.062091 5221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:08.066550 5221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:08.067180 5221 catalog_manager.cc:1259] Loaded cluster ID: 61aead23013f41c9b608f3969d91ece0
I20260809 09:31:08.067246 5221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:08.068949 5221 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:08.069491 5221 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Loaded TSK: 0
I20260809 09:31:08.069578 5221 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 24691abac8754f36bd7a6146d48893d3: Loaded TSK: 1
F20260809 09:31:08.070132 5221 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7effb10d19ad google::LogMessage::Fail() at ??:0
@ 0x7effb10d488a google::LogMessage::SendToLog() at ??:0
@ 0x7effb10d14de google::LogMessage::Flush() at ??:0
@ 0x7effb10d2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7effb3d9518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7effb3d8df0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7effbba3a7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7effbba1499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7effbba15178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7effbba56df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7effbba7582b std::function<>::operator()() at ??:0
@ 0x7effbba14b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7effbba15f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7effbba12579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7effbba56640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7effb33233d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7effb3323c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7effb33253f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7effb3314512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7effb49586db start_thread at ??:0
@ 0x7effafbcf61f clone at ??:0
W20260809 09:31:08.154170 5196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:08.154450 5196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:08.154510 5196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:08.160490 5196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:08.160586 5196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:08.160679 5196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:31:08.167680 5196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:38573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=42599
--tserver_master_addrs=127.9.102.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:08.168761 5196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:08.169919 5196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:08.179962 5281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:08.179957 5282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:08.180133 5284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:08.180366 5196 server_base.cc:1034] running on GCE node
I20260809 09:31:08.180717 5196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:08.181344 5196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:08.182515 5196 hybrid_clock.cc:648] HybridClock initialized: now 1786267868182498 us; error 33 us; skew 500 ppm
Aug 09 09:31:08 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:08.185048 5196 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:31:08.185791 5196 webserver.cc:460] Webserver started at http://127.9.102.1:42599/ using document root <none> and password file <none>
I20260809 09:31:08.186277 5196 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:08.186332 5196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:08.189599 5196 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260809 09:31:08.191598 5294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:08.192750 5196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 09:31:08.192874 5196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "e1737b2a81854cf0ad1384f1e7538922"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "24eef352574eab1027b6b62646d1a9d9"
server_key_iv: "9d7c5b318c1f8738cfbc7fb93bbc3d07"
server_key_version: "encryptionkey@0"
I20260809 09:31:08.193380 5196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:08.206326 5196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:08.208916 5196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:08.209093 5196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:08.209579 5196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:08.210510 5305 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260809 09:31:08.215355 5196 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 09:31:08.215421 5196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260809 09:31:08.215505 5196 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 09:31:08.216964 5196 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 09:31:08.217028 5196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260809 09:31:08.217320 5305 tablet_bootstrap.cc:492] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: Bootstrap starting.
I20260809 09:31:08.222968 5305 log.cc:824] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:08.226231 5196 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:38573
I20260809 09:31:08.226908 5196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:31:08.227851 5484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:38573 every 8 connection(s)
I20260809 09:31:08.231467 5305 tablet_bootstrap.cc:492] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 09:31:08.231834 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 5196
I20260809 09:31:08.231889 5305 tablet_bootstrap.cc:492] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: Bootstrap complete.
I20260809 09:31:08.232245 5305 ts_tablet_manager.cc:1397] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260809 09:31:08.232371 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:34269
--local_ip_for_outbound_sockets=127.9.102.2
--tserver_master_addrs=127.9.102.62:40855
--webserver_port=32933
--webserver_interface=127.9.102.2
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:31:08.235088 5305 raft_consensus.cc:348] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:31:08.235634 5305 raft_consensus.cc:729] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1737b2a81854cf0ad1384f1e7538922, State: Initialized, Role: FOLLOWER
I20260809 09:31:08.236207 5305 consensus_queue.cc:260] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:31:08.237196 5305 ts_tablet_manager.cc:1428] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260809 09:31:08.291260 5489 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:31:08.228570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:39181)
0809 09:31:08.231817 (+ 3247us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:08.231867 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:08.232991 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:08.290614 (+ 57623us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3066,"thread_start_us":87,"threads_started":1}
W20260809 09:31:08.291787 5485 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:40855 (0 consecutive failures): Network error: Failed to ping master at 127.9.102.62:40855: Client connection negotiation failed: client connection to 127.9.102.62:40855: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 09:31:08.292539 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:08.292305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:35773)
0809 09:31:08.292424 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:08.292484 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 09:31:08.293294 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:08.293038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:50085)
0809 09:31:08.293179 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:08.293248 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260809 09:31:08.293519 5485 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 09:31:08.337296 5490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:08.337523 5490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:08.337616 5490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:08.341938 5490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:08.341992 5490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:08.342092 5490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:31:08.347606 5490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:34269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=32933
--tserver_master_addrs=127.9.102.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:08.348364 5490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:08.349201 5490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:08.357096 5512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:08.357092 5507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:08.357146 5490 server_base.cc:1034] running on GCE node
W20260809 09:31:08.357092 5508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:08.357739 5490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:08.358270 5490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:08.359445 5490 hybrid_clock.cc:648] HybridClock initialized: now 1786267868359428 us; error 28 us; skew 500 ppm
Aug 09 09:31:08 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:08.361927 5490 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:31:08.362668 5490 webserver.cc:460] Webserver started at http://127.9.102.2:32933/ using document root <none> and password file <none>
I20260809 09:31:08.363142 5490 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:08.363229 5490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:08.366429 5490 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:08.368063 5528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:08.369174 5490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:08.369391 5490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "9eb88197ab8e47f1979f0b882e18af33"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "190b53b1c8fc7622a603534948d7cdb6"
server_key_iv: "6727519ec2395abd46ce8d04881c6aa1"
server_key_version: "encryptionkey@0"
I20260809 09:31:08.369839 5490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:08.386864 5490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:08.389178 5490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:08.389356 5490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:08.389822 5490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:08.390476 5490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:08.390547 5490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:08.390607 5490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:08.390637 5490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:08.399142 5490 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:34269
I20260809 09:31:08.399181 5707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:34269 every 8 connection(s)
I20260809 09:31:08.399989 5490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:31:08.403293 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:08.401777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:46557)
0809 09:31:08.402179 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:08.402272 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
W20260809 09:31:08.403743 5708 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:40855 (0 consecutive failures): Network error: Failed to ping master at 127.9.102.62:40855: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
I20260809 09:31:08.404323 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:08.404081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:57615)
0809 09:31:08.404232 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:08.404275 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 09:31:08.404917 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:08.404752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:52125)
0809 09:31:08.404835 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:08.404873 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260809 09:31:08.405104 5708 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 09:31:08.409482 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 5490
I20260809 09:31:08.409904 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:46267
--local_ip_for_outbound_sockets=127.9.102.3
--tserver_master_addrs=127.9.102.62:40855
--webserver_port=33889
--webserver_interface=127.9.102.3
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:08.513234 5711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:08.513437 5711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:08.513473 5711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:08.516839 5711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:08.516995 5711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:08.517132 5711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:31:08.521822 5711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:46267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=33889
--tserver_master_addrs=127.9.102.62:40855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:08.522665 5711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:08.523481 5711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:08.531888 5720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:08.531903 5725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:08.531888 5721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:08.532248 5711 server_base.cc:1034] running on GCE node
I20260809 09:31:08.532612 5711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:08.533169 5711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:08.534363 5711 hybrid_clock.cc:648] HybridClock initialized: now 1786267868534340 us; error 40 us; skew 500 ppm
Aug 09 09:31:08 dist-test-slave-7vjv krb5kdc[3187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:08.536816 5711 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:31:08.537647 5711 webserver.cc:460] Webserver started at http://127.9.102.3:33889/ using document root <none> and password file <none>
I20260809 09:31:08.538134 5711 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:08.538226 5711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:08.541579 5711 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260809 09:31:08.543349 5734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:08.544577 5711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 09:31:08.544698 5711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "82dd3517f63f4ee2aabdcc6e0716c097"
format_stamp: "Formatted at 2026-08-09 09:30:26 on dist-test-slave-7vjv"
server_key: "bd3c7a4fab525e914c9e8e4661c6cd7a"
server_key_iv: "90d1c3fa1843073a8d575c31f63fdf1c"
server_key_version: "encryptionkey@0"
I20260809 09:31:08.545118 5711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:08.558410 5711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:08.560755 5711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:08.560922 5711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:08.561443 5711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:08.562244 5711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:08.562314 5711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:08.562378 5711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:08.562418 5711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:08.570976 5711 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:46267
I20260809 09:31:08.570996 5918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:46267 every 8 connection(s)
I20260809 09:31:08.571805 5711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:31:08.575248 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:08.573603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:41995)
0809 09:31:08.574096 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:08.574190 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":139,"threads_started":1}
W20260809 09:31:08.575654 5919 heartbeater.cc:644] Failed to heartbeat to 127.9.102.62:40855 (0 consecutive failures): Network error: Failed to ping master at 127.9.102.62:40855: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
I20260809 09:31:08.576192 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:08.575973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:52037)
0809 09:31:08.576063 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:08.576143 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 09:31:08.576799 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:08.576589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:48399)
0809 09:31:08.576687 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:08.576740 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260809 09:31:08.576915 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 5711
W20260809 09:31:08.576983 5919 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 09:31:09.294484 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:09.294091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:34501)
0809 09:31:09.294341 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:09.294392 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
I20260809 09:31:09.405388 5930 raft_consensus.cc:482] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:31:09.405607 5930 raft_consensus.cc:504] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:31:09.406008 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:09.405658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:38695)
0809 09:31:09.405881 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:09.405926 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":76,"threads_started":1}
I20260809 09:31:09.406726 5930 leader_election.cc:288] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267), 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269)
Aug 09 09:31:09 dist-test-slave-7vjv krb5kdc[3187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.2@KRBTEST.COM
Aug 09 09:31:09 dist-test-slave-7vjv krb5kdc[3187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267868, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.3@KRBTEST.COM
I20260809 09:31:09.420632 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:09.407024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:33919 (local address 127.9.102.2:34269)
0809 09:31:09.407422 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:09.407443 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:09.407483 (+ 40us) server_negotiation.cc:407] Connection header received
0809 09:31:09.408156 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:09.408161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:09.408454 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:09.408722 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:09.409985 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:09.411021 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:09.412411 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:09.412904 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:09.413102 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:09.415979 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:09.415984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:09.415994 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:09.416019 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:09.418145 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:09.418626 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:09.418630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:09.418634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:09.418773 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:09.419094 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:09.419096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:09.419098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:09.419445 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:09.419602 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:09.420202 (+ 600us) server_negotiation.cc:299] Negotiation successful
0809 09:31:09.420449 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":66,"threads_started":1}
I20260809 09:31:09.420616 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:09.406887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:46267 (local address 127.9.102.1:56649)
0809 09:31:09.407084 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:09.407099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:09.407196 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:09.408813 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:09.408818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:09.409339 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:09.409825 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:09.409839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:09.411265 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:09.411267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:09.412233 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:09.412363 (+ 130us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:09.413028 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:09.413032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:09.413305 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:09.413894 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:31:09.413909 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:09.415807 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:09.418009 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:09.418018 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:09.418033 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:09.418385 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:09.418767 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:09.418770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:09.418772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:09.418959 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:09.419572 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:09.419579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:09.419853 (+ 274us) client_negotiation.cc:769] Sending connection context
0809 09:31:09.420048 (+ 195us) client_negotiation.cc:240] Negotiation successful
0809 09:31:09.420380 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":201,"spinlock_wait_cycles":11136}
I20260809 09:31:09.420746 5935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:09.406933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34269 (local address 127.9.102.1:33919)
0809 09:31:09.407182 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:09.407194 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:09.407271 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:09.408813 (+ 1542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:09.408818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:09.409340 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:09.409825 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:09.409838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:09.411168 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:09.411172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:09.412070 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:09.412080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:09.413074 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:09.413077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:09.413304 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:09.413950 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:31:09.413961 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:09.415455 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:09.418257 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:09.418259 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:09.418260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:09.418509 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:09.418896 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:09.418899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:09.418900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:09.418976 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:09.419572 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:09.419579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:09.419848 (+ 269us) client_negotiation.cc:769] Sending connection context
0809 09:31:09.420047 (+ 199us) client_negotiation.cc:240] Negotiation successful
0809 09:31:09.420476 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":264,"spinlock_wait_cycles":17408,"thread_start_us":65,"threads_started":1}
I20260809 09:31:09.420686 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:09.407211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:56649 (local address 127.9.102.3:46267)
0809 09:31:09.407556 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:09.407566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:09.407595 (+ 29us) server_negotiation.cc:407] Connection header received
0809 09:31:09.408218 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:09.408224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:09.408500 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:09.408811 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:09.409997 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:09.411140 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:09.412486 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:09.412900 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:09.413101 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:09.415980 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:09.415985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:09.415994 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:09.416028 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:09.417875 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:09.418509 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:09.418515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:09.418520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:09.418669 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:09.419066 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:09.419070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:09.419073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:09.419446 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:09.419602 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:09.420204 (+ 602us) server_negotiation.cc:299] Negotiation successful
0809 09:31:09.420476 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":137,"threads_started":1}
W20260809 09:31:09.423314 5295 leader_election.cc:341] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267): Not found: Tablet not found: dd22c7388f1f4a71b8b8e6ece181b30d
W20260809 09:31:09.423667 5296 leader_election.cc:341] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269): Not found: Tablet not found: dd22c7388f1f4a71b8b8e6ece181b30d
I20260809 09:31:09.423774 5296 leader_election.cc:302] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1737b2a81854cf0ad1384f1e7538922; no voters: 82dd3517f63f4ee2aabdcc6e0716c097, 9eb88197ab8e47f1979f0b882e18af33
I20260809 09:31:09.424006 5930 raft_consensus.cc:2726] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 09:31:09.577800 5939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:09.577423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:40275)
0809 09:31:09.577645 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:09.577689 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260809 09:31:10.295573 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:10.295213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:38647)
0809 09:31:10.295430 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:10.295476 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260809 09:31:10.407063 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:10.406719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:35431)
0809 09:31:10.406944 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:10.406991 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":64,"threads_started":1}
I20260809 09:31:10.578895 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:10.578445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:50623)
0809 09:31:10.578761 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:10.578811 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":71,"threads_started":1}
I20260809 09:31:11.296836 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:11.296436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:60893)
0809 09:31:11.296699 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:11.296748 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260809 09:31:11.408239 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:11.407779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:46063)
0809 09:31:11.408072 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:11.408169 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":63,"threads_started":1}
I20260809 09:31:11.481386 5948 raft_consensus.cc:482] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:31:11.481511 5948 raft_consensus.cc:504] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:31:11.481863 5948 leader_election.cc:288] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267), 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269)
W20260809 09:31:11.482640 5296 leader_election.cc:341] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269): Not found: Tablet not found: dd22c7388f1f4a71b8b8e6ece181b30d
W20260809 09:31:11.482746 5295 leader_election.cc:341] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267): Not found: Tablet not found: dd22c7388f1f4a71b8b8e6ece181b30d
I20260809 09:31:11.482878 5295 leader_election.cc:302] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1737b2a81854cf0ad1384f1e7538922; no voters: 82dd3517f63f4ee2aabdcc6e0716c097, 9eb88197ab8e47f1979f0b882e18af33
I20260809 09:31:11.483016 5948 raft_consensus.cc:2726] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 09:31:11.580060 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:11.579731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:43201)
0809 09:31:11.579937 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:11.579985 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":61,"threads_started":1}
I20260809 09:31:12.297801 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:12.297454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:40129)
0809 09:31:12.297663 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:12.297709 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260809 09:31:12.409222 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:12.408870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:33363)
0809 09:31:12.409092 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:12.409139 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260809 09:31:12.581187 5960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:12.580767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:56161)
0809 09:31:12.581001 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:12.581096 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20260809 09:31:13.298952 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:13.298554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:32853)
0809 09:31:13.298811 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:13.298861 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":70,"threads_started":1}
I20260809 09:31:13.410375 5963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:13.410050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:59735)
0809 09:31:13.410261 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:13.410307 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260809 09:31:13.582312 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:13.581910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:47817)
0809 09:31:13.582173 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:13.582232 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
I20260809 09:31:13.620281 5966 raft_consensus.cc:482] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:31:13.620412 5966 raft_consensus.cc:504] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:31:13.620781 5966 leader_election.cc:288] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267), 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269)
W20260809 09:31:13.621466 5296 leader_election.cc:341] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269): Not found: Tablet not found: dd22c7388f1f4a71b8b8e6ece181b30d
W20260809 09:31:13.621565 5295 leader_election.cc:341] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267): Not found: Tablet not found: dd22c7388f1f4a71b8b8e6ece181b30d
I20260809 09:31:13.621619 5295 leader_election.cc:302] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1737b2a81854cf0ad1384f1e7538922; no voters: 82dd3517f63f4ee2aabdcc6e0716c097, 9eb88197ab8e47f1979f0b882e18af33
I20260809 09:31:13.621748 5966 raft_consensus.cc:2726] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 09:31:14.300024 5968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:14.299646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:55193)
0809 09:31:14.299878 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:14.299929 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260809 09:31:14.411487 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:14.411097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:41149)
0809 09:31:14.411349 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:14.411410 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
I20260809 09:31:14.583339 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:14.582970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:40813)
0809 09:31:14.583224 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:14.583272 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":98,"threads_started":1}
I20260809 09:31:15.301102 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:15.300700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:55781)
0809 09:31:15.300928 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:15.300996 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20260809 09:31:15.412662 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:15.412266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:45601)
0809 09:31:15.412538 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:15.412589 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260809 09:31:15.584595 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:15.584199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:49723)
0809 09:31:15.584457 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:15.584517 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":61,"threads_started":1}
I20260809 09:31:16.302232 5975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:16.301881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:45907)
0809 09:31:16.302096 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:16.302143 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":65,"threads_started":1}
I20260809 09:31:16.381652 5976 raft_consensus.cc:482] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:31:16.381767 5976 raft_consensus.cc:504] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:31:16.382113 5976 leader_election.cc:288] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267), 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269)
W20260809 09:31:16.382790 5295 leader_election.cc:341] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267): Not found: Tablet not found: dd22c7388f1f4a71b8b8e6ece181b30d
W20260809 09:31:16.382879 5296 leader_election.cc:341] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269): Not found: Tablet not found: dd22c7388f1f4a71b8b8e6ece181b30d
I20260809 09:31:16.382953 5296 leader_election.cc:302] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1737b2a81854cf0ad1384f1e7538922; no voters: 82dd3517f63f4ee2aabdcc6e0716c097, 9eb88197ab8e47f1979f0b882e18af33
I20260809 09:31:16.383081 5976 raft_consensus.cc:2726] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 09:31:16.413585 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:16.413247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:50799)
0809 09:31:16.413470 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:16.413517 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":72,"threads_started":1}
I20260809 09:31:16.585562 5985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:16.585217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:49287)
0809 09:31:16.585435 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:16.585487 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":60,"threads_started":1}
I20260809 09:31:17.303400 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:17.302999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:34297)
0809 09:31:17.303259 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:17.303309 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
I20260809 09:31:17.414515 5987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:17.414188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:43755)
0809 09:31:17.414394 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:17.414440 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":63,"threads_started":1}
I20260809 09:31:17.586511 5988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:17.586174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:36809)
0809 09:31:17.586389 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:17.586438 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
I20260809 09:31:18.304641 5990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:18.304198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:33531)
0809 09:31:18.304493 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:18.304550 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":122,"threads_started":1}
I20260809 09:31:18.415498 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:18.415167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:34271)
0809 09:31:18.415381 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:18.415425 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20260809 09:31:18.587587 5994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:18.587134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:37943)
0809 09:31:18.587446 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:18.587497 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":64,"threads_started":1}
I20260809 09:31:19.305660 5997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:19.305304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:43799)
0809 09:31:19.305522 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:19.305570 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
I20260809 09:31:19.416872 5998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:19.416403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:41217)
0809 09:31:19.416702 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:19.416767 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
I20260809 09:31:19.588583 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:19.588244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:52295)
0809 09:31:19.588445 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:19.588491 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260809 09:31:20.306684 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:20.306309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:38035)
0809 09:31:20.306521 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:20.306596 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":68,"threads_started":1}
I20260809 09:31:20.417907 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:20.417589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:37099)
0809 09:31:20.417790 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:20.417835 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":61,"threads_started":1}
I20260809 09:31:20.589689 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:20.589339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:36499)
0809 09:31:20.589566 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:20.589616 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260809 09:31:21.307789 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:21.307346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:32863)
0809 09:31:21.307649 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:21.307699 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":83,"threads_started":1}
I20260809 09:31:21.419135 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:21.418762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:43483)
0809 09:31:21.418999 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:21.419056 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":67,"threads_started":1}
I20260809 09:31:21.590777 6009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:21.590454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:48011)
0809 09:31:21.590655 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:21.590701 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260809 09:31:22.308833 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:22.308445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:53495)
0809 09:31:22.308688 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:22.308739 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260809 09:31:22.420197 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:22.419785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:60159)
0809 09:31:22.420007 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:22.420064 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
I20260809 09:31:22.591886 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:22.591543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:45085)
0809 09:31:22.591760 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:22.591808 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260809 09:31:23.310047 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:23.309702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.1:56753)
0809 09:31:23.309907 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:23.309952 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
I20260809 09:31:23.421195 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:23.420883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.2:55447)
0809 09:31:23.421083 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:23.421126 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
I20260809 09:31:23.456858 6019 raft_consensus.cc:482] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:31:23.456966 6019 raft_consensus.cc:504] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82dd3517f63f4ee2aabdcc6e0716c097" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 46267 } } peers { permanent_uuid: "9eb88197ab8e47f1979f0b882e18af33" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34269 } } peers { permanent_uuid: "e1737b2a81854cf0ad1384f1e7538922" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38573 } }
I20260809 09:31:23.457297 6019 leader_election.cc:288] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267), 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269)
W20260809 09:31:23.457942 5295 leader_election.cc:341] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 82dd3517f63f4ee2aabdcc6e0716c097 (127.9.102.3:46267): Not found: Tablet not found: dd22c7388f1f4a71b8b8e6ece181b30d
W20260809 09:31:23.458043 5296 leader_election.cc:341] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9eb88197ab8e47f1979f0b882e18af33 (127.9.102.2:34269): Not found: Tablet not found: dd22c7388f1f4a71b8b8e6ece181b30d
I20260809 09:31:23.458101 5296 leader_election.cc:302] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1737b2a81854cf0ad1384f1e7538922; no voters: 82dd3517f63f4ee2aabdcc6e0716c097, 9eb88197ab8e47f1979f0b882e18af33
I20260809 09:31:23.458225 6019 raft_consensus.cc:2726] T dd22c7388f1f4a71b8b8e6ece181b30d P e1737b2a81854cf0ad1384f1e7538922 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 09:31:23.579453 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 5196
I20260809 09:31:23.593875 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:23.593491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40855 (local address 127.9.102.3:50325)
0809 09:31:23.593738 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:23.593786 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.62:40855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260809 09:31:23.598904 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 5490
I20260809 09:31:23.606048 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 5711
2026-08-09T09:31:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59126 ms)
[----------] 30 tests from SecurityITest (236923 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:23 dist-test-slave-7vjv krb5kdc[6024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:23 dist-test-slave-7vjv krb5kdc[6024](info): set up 2 sockets
Aug 09 09:31:23 dist-test-slave-7vjv krb5kdc[6024](info): commencing operation
krb5kdc: starting...
W20260809 09:31:24.787510 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:24 dist-test-slave-7vjv krb5kdc[6024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:31:24.898460 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40197
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:40197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:25.000727 6043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:25.000962 6043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:25.000998 6043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:25.004894 6043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:25.004944 6043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:25.004963 6043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:25.004978 6043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:25.004992 6043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:25.009773 6043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:40197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:25.010695 6043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:25.011590 6043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:25.019706 6057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:25.019721 6053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:25.019723 6054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:25.019886 6043 server_base.cc:1034] running on GCE node
I20260809 09:31:25.020612 6043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:25.021514 6043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:25.022696 6043 hybrid_clock.cc:648] HybridClock initialized: now 1786267885022677 us; error 29 us; skew 500 ppm
Aug 09 09:31:25 dist-test-slave-7vjv krb5kdc[6024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:25.025296 6043 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:31:25.026064 6043 webserver.cc:460] Webserver started at http://127.9.102.62:40161/ using document root <none> and password file <none>
I20260809 09:31:25.026609 6043 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:25.026665 6043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:25.026885 6043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:25.028808 6043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "095e83d40fde405d8f7126dfa36c77fa"
format_stamp: "Formatted at 2026-08-09 09:31:25 on dist-test-slave-7vjv"
server_key: "e35cc5d58ddfdda5b1715e60aa6bc76d"
server_key_iv: "62bed8ed6b58fdc1fb4f691b30846b00"
server_key_version: "encryptionkey@0"
I20260809 09:31:25.029333 6043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "095e83d40fde405d8f7126dfa36c77fa"
format_stamp: "Formatted at 2026-08-09 09:31:25 on dist-test-slave-7vjv"
server_key: "e35cc5d58ddfdda5b1715e60aa6bc76d"
server_key_iv: "62bed8ed6b58fdc1fb4f691b30846b00"
server_key_version: "encryptionkey@0"
I20260809 09:31:25.032891 6043 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:31:25.034946 6066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:25.036152 6043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:31:25.036281 6043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "095e83d40fde405d8f7126dfa36c77fa"
format_stamp: "Formatted at 2026-08-09 09:31:25 on dist-test-slave-7vjv"
server_key: "e35cc5d58ddfdda5b1715e60aa6bc76d"
server_key_iv: "62bed8ed6b58fdc1fb4f691b30846b00"
server_key_version: "encryptionkey@0"
I20260809 09:31:25.036381 6043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:25.050889 6043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:25.053331 6043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:25.053511 6043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:25.059883 6043 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:40197
I20260809 09:31:25.059966 6162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:40197 every 8 connection(s)
I20260809 09:31:25.060781 6043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:25.063269 6164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:25.063884 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 6043
I20260809 09:31:25.064014 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:25.064348 9624 external_mini_cluster.cc:1428] Setting key c976efffa7f5f78f9b5b744a8041ed47
I20260809 09:31:25.069394 6164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa: Bootstrap starting.
I20260809 09:31:25.071511 6164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:25.072422 6164 log.cc:824] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa: Log is configured to *not* fsync() on all Append() calls
Aug 09 09:31:25 dist-test-slave-7vjv krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:25.074155 6164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa: No bootstrap required, opened a new log
I20260809 09:31:25.077366 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.065652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.9.102.62:40197)
0809 09:31:25.066037 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:25.066050 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:25.066080 (+ 30us) server_negotiation.cc:407] Connection header received
0809 09:31:25.066672 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:25.066687 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:25.066958 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:25.067258 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:25.068040 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.068989 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.069918 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.070357 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.070527 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.073014 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:25.073022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:25.073037 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:25.073070 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:25.074747 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:25.075218 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:25.075222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:25.075226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:25.075330 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:25.075607 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:25.075609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:25.075610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:25.075939 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:25.076080 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:25.076432 (+ 352us) server_negotiation.cc:299] Negotiation successful
0809 09:31:25.076716 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":108,"threads_started":1}
I20260809 09:31:25.077973 6164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095e83d40fde405d8f7126dfa36c77fa" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40197 } }
I20260809 09:31:25.078220 6164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:25.078287 6164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095e83d40fde405d8f7126dfa36c77fa, State: Initialized, Role: FOLLOWER
I20260809 09:31:25.078701 6164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095e83d40fde405d8f7126dfa36c77fa" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40197 } }
I20260809 09:31:25.078862 6164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:25.078922 6164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:25.079011 6164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:25.079879 6164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095e83d40fde405d8f7126dfa36c77fa" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40197 } }
I20260809 09:31:25.080217 6164 leader_election.cc:302] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 095e83d40fde405d8f7126dfa36c77fa; no voters:
I20260809 09:31:25.080452 6164 leader_election.cc:288] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:25.080590 6173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:25.080840 6173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [term 1 LEADER]: Becoming Leader. State: Replica: 095e83d40fde405d8f7126dfa36c77fa, State: Running, Role: LEADER
I20260809 09:31:25.081148 6173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095e83d40fde405d8f7126dfa36c77fa" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40197 } }
I20260809 09:31:25.081637 6164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:25.082700 6176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "095e83d40fde405d8f7126dfa36c77fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095e83d40fde405d8f7126dfa36c77fa" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40197 } } }
I20260809 09:31:25.082832 6176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:25.083217 6192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:25.083623 6177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 095e83d40fde405d8f7126dfa36c77fa. Latest consensus state: current_term: 1 leader_uuid: "095e83d40fde405d8f7126dfa36c77fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095e83d40fde405d8f7126dfa36c77fa" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40197 } } }
I20260809 09:31:25.083811 6177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:25.085637 6192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:25.090687 6192 catalog_manager.cc:1347] Generated new cluster ID: 102c0b51706948c68e892781449652c6
I20260809 09:31:25.090770 6192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:25.098183 6192 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:25.099093 6192 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:25.106647 6192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa: Generated new TSK 0
I20260809 09:31:25.107313 6192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:31:25.135921 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40197
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:25.236594 6211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:25.236794 6211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:25.236828 6211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:25.240399 6211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:25.240444 6211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:25.240495 6211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:31:25.245029 6211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:25.245797 6211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:25.246582 6211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:25.254958 6219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:25.254964 6217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:25.255077 6221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:25.255328 6211 server_base.cc:1034] running on GCE node
I20260809 09:31:25.255693 6211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:25.256297 6211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:25.257472 6211 hybrid_clock.cc:648] HybridClock initialized: now 1786267885257461 us; error 26 us; skew 500 ppm
Aug 09 09:31:25 dist-test-slave-7vjv krb5kdc[6024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:25.260032 6211 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:31:25.260828 6211 webserver.cc:460] Webserver started at http://127.9.102.1:34987/ using document root <none> and password file <none>
I20260809 09:31:25.261366 6211 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:25.261451 6211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:25.261660 6211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:25.263491 6211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "3fd72c20911a4c46a5bac4678d5e706a"
format_stamp: "Formatted at 2026-08-09 09:31:25 on dist-test-slave-7vjv"
server_key: "cbb520b645b4fb6eb1d65b49176ef123"
server_key_iv: "5fa1e914372f16c33eca34e8b5b16ab6"
server_key_version: "encryptionkey@0"
I20260809 09:31:25.264032 6211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "3fd72c20911a4c46a5bac4678d5e706a"
format_stamp: "Formatted at 2026-08-09 09:31:25 on dist-test-slave-7vjv"
server_key: "cbb520b645b4fb6eb1d65b49176ef123"
server_key_iv: "5fa1e914372f16c33eca34e8b5b16ab6"
server_key_version: "encryptionkey@0"
I20260809 09:31:25.267428 6211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 09:31:25.269543 6234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:25.270594 6211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:31:25.270732 6211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "3fd72c20911a4c46a5bac4678d5e706a"
format_stamp: "Formatted at 2026-08-09 09:31:25 on dist-test-slave-7vjv"
server_key: "cbb520b645b4fb6eb1d65b49176ef123"
server_key_iv: "5fa1e914372f16c33eca34e8b5b16ab6"
server_key_version: "encryptionkey@0"
I20260809 09:31:25.270838 6211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:25.291066 6211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:25.293919 6211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:25.294119 6211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:25.294651 6211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:25.295423 6211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:25.295488 6211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:25.295549 6211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:25.295609 6211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:25.304065 6211 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:43911
I20260809 09:31:25.304087 6439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:43911 every 8 connection(s)
I20260809 09:31:25.304898 6211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:31:25.311177 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 6211
I20260809 09:31:25.311304 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:31:25.311627 9624 external_mini_cluster.cc:1428] Setting key e19f0a9c6f9ed1449bfc71633d44db09
Aug 09 09:31:25 dist-test-slave-7vjv krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:25.318218 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.306548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:60853 (local address 127.9.102.62:40197)
0809 09:31:25.306698 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:25.306703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:25.307375 (+ 672us) server_negotiation.cc:407] Connection header received
0809 09:31:25.308503 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:25.308506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:25.308549 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:25.308638 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:25.309628 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.310114 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.311159 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.311520 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.311626 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.314467 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:25.314473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:25.314478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:25.314511 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:25.315919 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:25.316588 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:25.316594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:25.316598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:25.316693 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:25.317024 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:25.317030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:25.317035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:25.317277 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:25.317380 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:25.317814 (+ 434us) server_negotiation.cc:299] Negotiation successful
0809 09:31:25.317972 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 09:31:25.318679 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.306845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40197 (local address 127.9.102.1:60853)
0809 09:31:25.307240 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:25.307276 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:25.308328 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:25.308793 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:25.308799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:25.309105 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:25.309502 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:25.309513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.310216 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.310218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:25.311039 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:25.311048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.311668 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.311672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:25.311831 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.312799 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:31:25.312814 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:25.314308 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:25.316065 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:25.316073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:25.316086 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:25.316464 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:25.316786 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:25.316788 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:25.316789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:25.316912 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:25.317377 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:25.317381 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:25.317582 (+ 201us) client_negotiation.cc:769] Sending connection context
0809 09:31:25.317712 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 09:31:25.317917 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":109,"threads_started":1}
I20260809 09:31:25.319818 6441 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40197
I20260809 09:31:25.320070 6441 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:25.320623 6441 heartbeater.cc:511] Master 127.9.102.62:40197 requested a full tablet report, sending...
I20260809 09:31:25.322299 6091 ts_manager.cc:194] Registered new tserver with Master: 3fd72c20911a4c46a5bac4678d5e706a (127.9.102.1:43911)
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:31:25.323612 6091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:60853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:31:25.339304 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40197
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:25.438839 6451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:25.439069 6451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:25.439129 6451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:25.442799 6451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:25.442847 6451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:25.442938 6451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:31:25.447376 6451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:25.448277 6451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:25.449059 6451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:25.457549 6470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:25.457454 6467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:25.457454 6472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:25.457713 6451 server_base.cc:1034] running on GCE node
I20260809 09:31:25.458397 6451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:25.458949 6451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:25.460145 6451 hybrid_clock.cc:648] HybridClock initialized: now 1786267885460132 us; error 26 us; skew 500 ppm
Aug 09 09:31:25 dist-test-slave-7vjv krb5kdc[6024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:25.462735 6451 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:31:25.463507 6451 webserver.cc:460] Webserver started at http://127.9.102.2:46253/ using document root <none> and password file <none>
I20260809 09:31:25.464038 6451 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:25.464154 6451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:25.464377 6451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:25.466101 6451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "e682caec54204c9cb88b1c5ace37101c"
format_stamp: "Formatted at 2026-08-09 09:31:25 on dist-test-slave-7vjv"
server_key: "fbc97dc81657adcc957a9df679743232"
server_key_iv: "4ea74dbd1ae1a234e2d3a5d0681eaead"
server_key_version: "encryptionkey@0"
I20260809 09:31:25.466599 6451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "e682caec54204c9cb88b1c5ace37101c"
format_stamp: "Formatted at 2026-08-09 09:31:25 on dist-test-slave-7vjv"
server_key: "fbc97dc81657adcc957a9df679743232"
server_key_iv: "4ea74dbd1ae1a234e2d3a5d0681eaead"
server_key_version: "encryptionkey@0"
I20260809 09:31:25.469836 6451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:31:25.471923 6480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:25.472990 6451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:25.473114 6451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "e682caec54204c9cb88b1c5ace37101c"
format_stamp: "Formatted at 2026-08-09 09:31:25 on dist-test-slave-7vjv"
server_key: "fbc97dc81657adcc957a9df679743232"
server_key_iv: "4ea74dbd1ae1a234e2d3a5d0681eaead"
server_key_version: "encryptionkey@0"
I20260809 09:31:25.473208 6451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:25.482549 6451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:25.485426 6451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:25.485601 6451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:25.486097 6451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:25.486862 6451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:25.486927 6451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:25.486989 6451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:25.487046 6451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:25.495599 6451 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:45123
I20260809 09:31:25.495692 6663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:45123 every 8 connection(s)
I20260809 09:31:25.496444 6451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:31:25.504858 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 6451
I20260809 09:31:25.504997 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
Aug 09 09:31:25 dist-test-slave-7vjv krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:25.505297 9624 external_mini_cluster.cc:1428] Setting key d1e357e23c7d87e6bf50b7dc535e1818
I20260809 09:31:25.509459 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.498026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:50693 (local address 127.9.102.62:40197)
0809 09:31:25.498154 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:25.498158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:25.498885 (+ 727us) server_negotiation.cc:407] Connection header received
0809 09:31:25.500234 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:25.500237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:25.500293 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:25.500400 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:25.501474 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.501963 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.503018 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.503241 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.503342 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.505636 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:25.505639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:25.505641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:25.505662 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:25.507171 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:25.507719 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:25.507721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:25.507722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:25.507804 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:25.508235 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:25.508237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:25.508238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:25.508409 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:25.508481 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:25.509081 (+ 600us) server_negotiation.cc:299] Negotiation successful
0809 09:31:25.509236 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:31:25.510098 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.498350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40197 (local address 127.9.102.2:50693)
0809 09:31:25.498744 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:25.498781 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:25.499981 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:25.500580 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:25.500586 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:25.500889 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:25.501327 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:25.501339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.502094 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.502099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:25.502891 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:25.502900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.503378 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.503382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:25.503556 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.503998 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:31:25.504012 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:25.505484 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:25.507280 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:25.507286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:25.507298 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:25.507630 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:25.507893 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:25.507898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:25.507902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:25.508112 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:25.508508 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:25.508516 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:25.508799 (+ 283us) client_negotiation.cc:769] Sending connection context
0809 09:31:25.508983 (+ 184us) client_negotiation.cc:240] Negotiation successful
0809 09:31:25.509332 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":110,"threads_started":1}
I20260809 09:31:25.511108 6664 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40197
I20260809 09:31:25.511328 6664 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:25.511737 6664 heartbeater.cc:511] Master 127.9.102.62:40197 requested a full tablet report, sending...
I20260809 09:31:25.512750 6091 ts_manager.cc:194] Registered new tserver with Master: e682caec54204c9cb88b1c5ace37101c (127.9.102.2:45123)
I20260809 09:31:25.513271 6091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:50693
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:31:25.531801 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40197
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:25.634079 6673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:25.634284 6673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:25.634320 6673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:25.638141 6673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:25.638193 6673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:25.638262 6673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:31:25.642899 6673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:40197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:25.643689 6673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:25.644601 6673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:25.653198 6682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:25.653247 6673 server_base.cc:1034] running on GCE node
W20260809 09:31:25.653183 6685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:25.653117 6688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:25.653787 6673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:25.654372 6673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:25.655572 6673 hybrid_clock.cc:648] HybridClock initialized: now 1786267885655542 us; error 44 us; skew 500 ppm
Aug 09 09:31:25 dist-test-slave-7vjv krb5kdc[6024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:25.658320 6673 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:31:25.659183 6673 webserver.cc:460] Webserver started at http://127.9.102.3:37451/ using document root <none> and password file <none>
I20260809 09:31:25.659698 6673 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:25.659758 6673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:25.660006 6673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:25.661767 6673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "6e4cbc653e094b7982e01b80732bc74e"
format_stamp: "Formatted at 2026-08-09 09:31:25 on dist-test-slave-7vjv"
server_key: "8422b0b12fc5e936bc2ad2afc69dc6ed"
server_key_iv: "62128431aa051cf6a5a54c4f23dafbc8"
server_key_version: "encryptionkey@0"
I20260809 09:31:25.662302 6673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "6e4cbc653e094b7982e01b80732bc74e"
format_stamp: "Formatted at 2026-08-09 09:31:25 on dist-test-slave-7vjv"
server_key: "8422b0b12fc5e936bc2ad2afc69dc6ed"
server_key_iv: "62128431aa051cf6a5a54c4f23dafbc8"
server_key_version: "encryptionkey@0"
I20260809 09:31:25.665629 6673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:31:25.667708 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:25.668819 6673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:25.668943 6673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "6e4cbc653e094b7982e01b80732bc74e"
format_stamp: "Formatted at 2026-08-09 09:31:25 on dist-test-slave-7vjv"
server_key: "8422b0b12fc5e936bc2ad2afc69dc6ed"
server_key_iv: "62128431aa051cf6a5a54c4f23dafbc8"
server_key_version: "encryptionkey@0"
I20260809 09:31:25.669034 6673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:25.684605 6673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:25.686966 6673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:25.687139 6673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:25.687628 6673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:25.688386 6673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:25.688431 6673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:25.688490 6673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:25.688536 6673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:25.696403 6673 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:33771
I20260809 09:31:25.696436 6900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:33771 every 8 connection(s)
I20260809 09:31:25.697122 6673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
Aug 09 09:31:25 dist-test-slave-7vjv krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267885, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:25.707257 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 6673
I20260809 09:31:25.707366 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:31:25.707620 9624 external_mini_cluster.cc:1428] Setting key ae089a9b05efc31c9600f885ecb7ecc7
I20260809 09:31:25.709292 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.698762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:51345 (local address 127.9.102.62:40197)
0809 09:31:25.698896 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:25.698899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:25.699622 (+ 723us) server_negotiation.cc:407] Connection header received
0809 09:31:25.700968 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:25.700971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:25.701027 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:25.701110 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:25.702115 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.702604 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.703574 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.703803 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.703932 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.706080 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:25.706083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:25.706085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:25.706105 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:25.707134 (+ 1029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:25.707645 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:25.707649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:25.707651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:25.707769 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:25.708249 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:25.708251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:25.708252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:25.708468 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:25.708578 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:25.709024 (+ 446us) server_negotiation.cc:299] Negotiation successful
0809 09:31:25.709143 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 09:31:25.710065 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.699080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:40197 (local address 127.9.102.3:51345)
0809 09:31:25.699455 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:25.699490 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:25.700761 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:25.701261 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:25.701266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:25.701579 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:25.701976 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:25.701987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.702732 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.702735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:25.703453 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:25.703461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.703962 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.703966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:25.704188 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.704637 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:31:25.704651 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:25.705915 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:25.707244 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:25.707248 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:25.707260 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:25.707554 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:25.707865 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:25.707868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:25.707870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:25.707998 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:25.708578 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:25.708585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:25.708809 (+ 224us) client_negotiation.cc:769] Sending connection context
0809 09:31:25.708991 (+ 182us) client_negotiation.cc:240] Negotiation successful
0809 09:31:25.709263 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":101,"threads_started":1}
I20260809 09:31:25.711019 6901 heartbeater.cc:347] Connected to a master server at 127.9.102.62:40197
I20260809 09:31:25.711269 6901 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:25.711688 6901 heartbeater.cc:511] Master 127.9.102.62:40197 requested a full tablet report, sending...
I20260809 09:31:25.712647 6091 ts_manager.cc:194] Registered new tserver with Master: 6e4cbc653e094b7982e01b80732bc74e (127.9.102.3:33771)
I20260809 09:31:25.713136 6091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:51345
I20260809 09:31:25.721167 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:31:25.729561 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.722892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.9.102.62:40197)
0809 09:31:25.723035 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:25.723040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:25.723211 (+ 171us) server_negotiation.cc:407] Connection header received
0809 09:31:25.723375 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:25.723378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:25.723416 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:25.723499 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:25.724153 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.724655 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.725416 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.725630 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.725730 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.726859 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:25.726862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:25.726865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:25.726887 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:25.727929 (+ 1042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:25.728384 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:25.728387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:25.728389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:25.728488 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:25.728765 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:25.728768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:25.728770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:25.728965 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:25.729108 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:25.729328 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 09:31:25.729429 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 09:31:25.731794 6091 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:31:25.734551 6091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:31:25.747998 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.742495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59808 (local address 127.9.102.2:45123)
0809 09:31:25.742910 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:25.742924 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:25.743262 (+ 338us) server_negotiation.cc:407] Connection header received
0809 09:31:25.743382 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:25.743387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:25.743580 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:25.743738 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:25.744089 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.744859 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.746524 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.747113 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.747215 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.747346 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:25.747758 (+ 412us) server_negotiation.cc:299] Negotiation successful
0809 09:31:25.747853 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":140,"threads_started":1}
I20260809 09:31:25.748018 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.742546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.9.102.3:33771)
0809 09:31:25.742910 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:25.742922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:25.743583 (+ 661us) server_negotiation.cc:407] Connection header received
0809 09:31:25.743662 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:25.743667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:25.743838 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:25.743975 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:25.744222 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.745259 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.746562 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.747276 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.747375 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.747492 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:25.747816 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 09:31:25.747899 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":96,"threads_started":1}
I20260809 09:31:25.747999 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.741929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:45123 (local address 127.0.0.1:59808)
0809 09:31:25.743169 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:25.743181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:25.743272 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:25.743857 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:25.743859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:25.743873 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:25.743966 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:25.743971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.745013 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.745016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:25.746387 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:25.746397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.747407 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.747411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:25.747546 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.747607 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 09:31:25.747711 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 09:31:25.747861 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"mutex_wait_us":97,"thread_start_us":86,"threads_started":1}
I20260809 09:31:25.748219 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.741919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42504 (local address 127.9.102.1:43911)
0809 09:31:25.742235 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:25.742247 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:25.742939 (+ 692us) server_negotiation.cc:407] Connection header received
0809 09:31:25.743023 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:25.743029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:25.743245 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:25.743395 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:25.743671 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.744845 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.746494 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.747276 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.747375 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.747492 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:25.747938 (+ 446us) server_negotiation.cc:299] Negotiation successful
0809 09:31:25.748066 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":151,"threads_started":1}
I20260809 09:31:25.748322 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.742412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:33771 (local address 127.0.0.1:42710)
0809 09:31:25.743454 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:25.743466 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:25.743551 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:25.743968 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:25.743970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:25.743983 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:25.744081 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:25.744087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.745400 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.745403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:25.746467 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:25.746475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.747449 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.747451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:25.747648 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.747675 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 09:31:25.747734 (+ 59us) client_negotiation.cc:240] Negotiation successful
0809 09:31:25.748209 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":505,"thread_start_us":94,"threads_started":1}
I20260809 09:31:25.748417 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:25.741772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:43911 (local address 127.0.0.1:42504)
0809 09:31:25.742609 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:25.742651 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:25.742791 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:25.743398 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:25.743403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:25.743427 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:25.743547 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:25.743556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.744987 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.744992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:25.746374 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:25.746383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:25.747407 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:25.747411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:25.747574 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:25.747659 (+ 85us) client_negotiation.cc:769] Sending connection context
0809 09:31:25.747749 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 09:31:25.747846 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":346,"thread_start_us":66,"threads_started":1}
I20260809 09:31:25.751019 6560 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0178d73ec140fb9ff9ecb688a04754 (DEFAULT_TABLE table=test-table [id=68ef3db1a8174f79b89016806789d25c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:25.751046 6784 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0178d73ec140fb9ff9ecb688a04754 (DEFAULT_TABLE table=test-table [id=68ef3db1a8174f79b89016806789d25c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:25.751053 6325 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0178d73ec140fb9ff9ecb688a04754 (DEFAULT_TABLE table=test-table [id=68ef3db1a8174f79b89016806789d25c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:25.751909 6560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0178d73ec140fb9ff9ecb688a04754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:25.751959 6325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0178d73ec140fb9ff9ecb688a04754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:25.751959 6784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a0178d73ec140fb9ff9ecb688a04754. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:25.757536 6929 tablet_bootstrap.cc:492] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c: Bootstrap starting.
I20260809 09:31:25.759255 6930 tablet_bootstrap.cc:492] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a: Bootstrap starting.
I20260809 09:31:25.760349 6929 tablet_bootstrap.cc:654] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:25.760392 6933 tablet_bootstrap.cc:492] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e: Bootstrap starting.
I20260809 09:31:25.761618 6929 log.cc:824] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:25.762481 6933 tablet_bootstrap.cc:654] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:25.762704 6930 tablet_bootstrap.cc:654] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:25.763337 6930 log.cc:824] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:25.763842 6933 log.cc:824] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:25.764940 6929 tablet_bootstrap.cc:492] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c: No bootstrap required, opened a new log
I20260809 09:31:25.765144 6929 ts_tablet_manager.cc:1397] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260809 09:31:25.765938 6933 tablet_bootstrap.cc:492] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e: No bootstrap required, opened a new log
I20260809 09:31:25.766116 6933 ts_tablet_manager.cc:1397] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260809 09:31:25.768045 6929 raft_consensus.cc:348] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd72c20911a4c46a5bac4678d5e706a" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43911 } } peers { permanent_uuid: "e682caec54204c9cb88b1c5ace37101c" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45123 } } peers { permanent_uuid: "6e4cbc653e094b7982e01b80732bc74e" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33771 } }
I20260809 09:31:25.768363 6929 raft_consensus.cc:374] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:25.768416 6929 raft_consensus.cc:729] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e682caec54204c9cb88b1c5ace37101c, State: Initialized, Role: FOLLOWER
I20260809 09:31:25.768831 6929 consensus_queue.cc:260] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd72c20911a4c46a5bac4678d5e706a" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43911 } } peers { permanent_uuid: "e682caec54204c9cb88b1c5ace37101c" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45123 } } peers { permanent_uuid: "6e4cbc653e094b7982e01b80732bc74e" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33771 } }
I20260809 09:31:25.769572 6930 tablet_bootstrap.cc:492] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a: No bootstrap required, opened a new log
I20260809 09:31:25.769789 6930 ts_tablet_manager.cc:1397] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260809 09:31:25.769789 6929 ts_tablet_manager.cc:1428] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:31:25.769846 6933 raft_consensus.cc:348] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd72c20911a4c46a5bac4678d5e706a" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43911 } } peers { permanent_uuid: "e682caec54204c9cb88b1c5ace37101c" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45123 } } peers { permanent_uuid: "6e4cbc653e094b7982e01b80732bc74e" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33771 } }
I20260809 09:31:25.770128 6933 raft_consensus.cc:374] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:25.770129 6664 heartbeater.cc:503] Master 127.9.102.62:40197 was elected leader, sending a full tablet report...
I20260809 09:31:25.770192 6933 raft_consensus.cc:729] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4cbc653e094b7982e01b80732bc74e, State: Initialized, Role: FOLLOWER
I20260809 09:31:25.770629 6933 consensus_queue.cc:260] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd72c20911a4c46a5bac4678d5e706a" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43911 } } peers { permanent_uuid: "e682caec54204c9cb88b1c5ace37101c" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45123 } } peers { permanent_uuid: "6e4cbc653e094b7982e01b80732bc74e" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33771 } }
I20260809 09:31:25.771229 6901 heartbeater.cc:503] Master 127.9.102.62:40197 was elected leader, sending a full tablet report...
I20260809 09:31:25.771497 6933 ts_tablet_manager.cc:1428] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260809 09:31:25.773885 6930 raft_consensus.cc:348] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd72c20911a4c46a5bac4678d5e706a" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43911 } } peers { permanent_uuid: "e682caec54204c9cb88b1c5ace37101c" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45123 } } peers { permanent_uuid: "6e4cbc653e094b7982e01b80732bc74e" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33771 } }
I20260809 09:31:25.774135 6930 raft_consensus.cc:374] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:25.774224 6930 raft_consensus.cc:729] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd72c20911a4c46a5bac4678d5e706a, State: Initialized, Role: FOLLOWER
I20260809 09:31:25.774639 6930 consensus_queue.cc:260] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd72c20911a4c46a5bac4678d5e706a" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43911 } } peers { permanent_uuid: "e682caec54204c9cb88b1c5ace37101c" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45123 } } peers { permanent_uuid: "6e4cbc653e094b7982e01b80732bc74e" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33771 } }
I20260809 09:31:25.775345 6930 ts_tablet_manager.cc:1428] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:31:25.775707 6441 heartbeater.cc:503] Master 127.9.102.62:40197 was elected leader, sending a full tablet report...
W20260809 09:31:25.805998 6444 tablet.cc:2369] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:31:25.947955 6902 tablet.cc:2369] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:31:25.997612 6666 tablet.cc:2369] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:31:26.038333 6939 raft_consensus.cc:482] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:31:26.038522 6939 raft_consensus.cc:504] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd72c20911a4c46a5bac4678d5e706a" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43911 } } peers { permanent_uuid: "e682caec54204c9cb88b1c5ace37101c" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45123 } } peers { permanent_uuid: "6e4cbc653e094b7982e01b80732bc74e" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33771 } }
I20260809 09:31:26.039579 6939 leader_election.cc:288] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fd72c20911a4c46a5bac4678d5e706a (127.9.102.1:43911), e682caec54204c9cb88b1c5ace37101c (127.9.102.2:45123)
I20260809 09:31:26.042991 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.039835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:43911 (local address 127.9.102.3:57961)
0809 09:31:26.039972 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:26.039988 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:26.040073 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:26.040358 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:26.040361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:26.040378 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:26.040490 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.040496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.041285 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.041289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.042005 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.042013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.042620 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.042623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.042715 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.042738 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 09:31:26.042815 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 09:31:26.042874 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260809 09:31:26.042991 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.039928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:57961 (local address 127.9.102.1:43911)
0809 09:31:26.040052 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:26.040056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:26.040071 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:31:26.040218 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:26.040221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:26.040271 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:26.040349 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:26.040623 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.041177 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.042122 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.042497 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.042616 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.042675 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:26.042833 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 09:31:26.042895 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260809 09:31:26.043234 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.039932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:32827 (local address 127.9.102.2:45123)
0809 09:31:26.040063 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:26.040066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:26.040507 (+ 441us) server_negotiation.cc:407] Connection header received
0809 09:31:26.040654 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:26.040657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:26.040706 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:26.040785 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:26.041045 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.041479 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.042416 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.042768 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.042872 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.042923 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:26.043085 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 09:31:26.043143 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 09:31:26.043249 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.039840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:45123 (local address 127.9.102.3:32827)
0809 09:31:26.040413 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:26.040424 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:26.040546 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:26.040810 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:26.040813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:26.040827 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:26.040937 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.040942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.041592 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.041595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.042297 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.042305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.042877 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.042880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.042964 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.042990 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 09:31:26.043069 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 09:31:26.043126 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":231,"thread_start_us":109,"threads_started":1}
I20260809 09:31:26.043648 6363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a0178d73ec140fb9ff9ecb688a04754" candidate_uuid: "6e4cbc653e094b7982e01b80732bc74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd72c20911a4c46a5bac4678d5e706a" is_pre_election: true
I20260809 09:31:26.043823 6590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a0178d73ec140fb9ff9ecb688a04754" candidate_uuid: "6e4cbc653e094b7982e01b80732bc74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e682caec54204c9cb88b1c5ace37101c" is_pre_election: true
I20260809 09:31:26.043931 6363 raft_consensus.cc:2454] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e4cbc653e094b7982e01b80732bc74e in term 0.
I20260809 09:31:26.044071 6590 raft_consensus.cc:2454] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e4cbc653e094b7982e01b80732bc74e in term 0.
I20260809 09:31:26.044533 6706 leader_election.cc:302] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3fd72c20911a4c46a5bac4678d5e706a, 6e4cbc653e094b7982e01b80732bc74e, e682caec54204c9cb88b1c5ace37101c; no voters:
I20260809 09:31:26.044786 6939 raft_consensus.cc:2781] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:31:26.044855 6939 raft_consensus.cc:482] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:31:26.044881 6939 raft_consensus.cc:3037] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:26.045856 6939 raft_consensus.cc:504] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd72c20911a4c46a5bac4678d5e706a" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43911 } } peers { permanent_uuid: "e682caec54204c9cb88b1c5ace37101c" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45123 } } peers { permanent_uuid: "6e4cbc653e094b7982e01b80732bc74e" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33771 } }
I20260809 09:31:26.046192 6939 leader_election.cc:288] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [CANDIDATE]: Term 1 election: Requested vote from peers 3fd72c20911a4c46a5bac4678d5e706a (127.9.102.1:43911), e682caec54204c9cb88b1c5ace37101c (127.9.102.2:45123)
I20260809 09:31:26.046487 6363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a0178d73ec140fb9ff9ecb688a04754" candidate_uuid: "6e4cbc653e094b7982e01b80732bc74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd72c20911a4c46a5bac4678d5e706a"
I20260809 09:31:26.046535 6590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a0178d73ec140fb9ff9ecb688a04754" candidate_uuid: "6e4cbc653e094b7982e01b80732bc74e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e682caec54204c9cb88b1c5ace37101c"
I20260809 09:31:26.046640 6363 raft_consensus.cc:3037] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:26.046675 6590 raft_consensus.cc:3037] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:26.047549 6363 raft_consensus.cc:2454] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e4cbc653e094b7982e01b80732bc74e in term 1.
I20260809 09:31:26.047667 6590 raft_consensus.cc:2454] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e4cbc653e094b7982e01b80732bc74e in term 1.
I20260809 09:31:26.047827 6705 leader_election.cc:302] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd72c20911a4c46a5bac4678d5e706a, 6e4cbc653e094b7982e01b80732bc74e; no voters:
I20260809 09:31:26.048003 6939 raft_consensus.cc:2781] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:26.048241 6939 raft_consensus.cc:686] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 1 LEADER]: Becoming Leader. State: Replica: 6e4cbc653e094b7982e01b80732bc74e, State: Running, Role: LEADER
I20260809 09:31:26.048542 6939 consensus_queue.cc:237] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd72c20911a4c46a5bac4678d5e706a" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43911 } } peers { permanent_uuid: "e682caec54204c9cb88b1c5ace37101c" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45123 } } peers { permanent_uuid: "6e4cbc653e094b7982e01b80732bc74e" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33771 } }
I20260809 09:31:26.051553 6089 catalog_manager.cc:5495] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e4cbc653e094b7982e01b80732bc74e (127.9.102.3). New cstate: current_term: 1 leader_uuid: "6e4cbc653e094b7982e01b80732bc74e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fd72c20911a4c46a5bac4678d5e706a" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e682caec54204c9cb88b1c5ace37101c" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e4cbc653e094b7982e01b80732bc74e" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33771 } health_report { overall_health: HEALTHY } } }
I20260809 09:31:26.107419 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.104191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.9.102.3:33771)
0809 09:31:26.104300 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:26.104305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:26.104340 (+ 35us) server_negotiation.cc:407] Connection header received
0809 09:31:26.104513 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:26.104515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:26.104564 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:26.104685 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:31:26.104915 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.105370 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.106179 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.106367 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.106463 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.106732 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:31:26.107065 (+ 333us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:31:26.107180 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:26.107245 (+ 65us) server_negotiation.cc:299] Negotiation successful
0809 09:31:26.107289 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 09:31:26.112150 6363 raft_consensus.cc:1261] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a [term 1 FOLLOWER]: Refusing update from remote peer 6e4cbc653e094b7982e01b80732bc74e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:31:26.112175 6590 raft_consensus.cc:1261] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c [term 1 FOLLOWER]: Refusing update from remote peer 6e4cbc653e094b7982e01b80732bc74e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:31:26.112775 6942 consensus_queue.cc:1035] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e682caec54204c9cb88b1c5ace37101c" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:31:26.112941 6939 consensus_queue.cc:1035] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd72c20911a4c46a5bac4678d5e706a" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 43911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:31:26.118714 6947 mvcc.cc:205] Tried to move back new op lower bound from 7316553261509394432 to 7316553261257248768. Current Snapshot: MvccSnapshot[applied={T|T < 7316553261509394432}]
I20260809 09:31:26.127328 6089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:31:26.127470 6089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:31:26.129590 6089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 095e83d40fde405d8f7126dfa36c77fa: Sending DeleteTablet for 3 replicas of tablet 7a0178d73ec140fb9ff9ecb688a04754
I20260809 09:31:26.130256 6325 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a0178d73ec140fb9ff9ecb688a04754 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:26 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:42504
I20260809 09:31:26.130379 6560 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a0178d73ec140fb9ff9ecb688a04754 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:26 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:59808
I20260809 09:31:26.130688 6784 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a0178d73ec140fb9ff9ecb688a04754 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:26 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:42710
I20260809 09:31:26.130880 6968 tablet_replica.cc:330] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c: stopping tablet replica
I20260809 09:31:26.130936 6967 tablet_replica.cc:330] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a: stopping tablet replica
I20260809 09:31:26.131227 6967 raft_consensus.cc:2229] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:31:26.131237 6968 raft_consensus.cc:2229] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:31:26.131500 6967 raft_consensus.cc:2258] T 7a0178d73ec140fb9ff9ecb688a04754 P 3fd72c20911a4c46a5bac4678d5e706a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:26.131603 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 6211
I20260809 09:31:26.131642 6968 raft_consensus.cc:2258] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:26.134126 6968 ts_tablet_manager.cc:1905] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:31:26.137341 6968 ts_tablet_manager.cc:1918] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:31:26.137471 6968 log.cc:1196] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/7a0178d73ec140fb9ff9ecb688a04754
I20260809 09:31:26.137807 6968 ts_tablet_manager.cc:1939] T 7a0178d73ec140fb9ff9ecb688a04754 P e682caec54204c9cb88b1c5ace37101c: Deleting consensus metadata
I20260809 09:31:26.140527 6068 catalog_manager.cc:4841] TS e682caec54204c9cb88b1c5ace37101c (127.9.102.2:45123): tablet 7a0178d73ec140fb9ff9ecb688a04754 (table test-table [id=68ef3db1a8174f79b89016806789d25c]) successfully deleted
I20260809 09:31:26.142120 6970 tablet_replica.cc:330] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e: stopping tablet replica
I20260809 09:31:26.142369 6970 raft_consensus.cc:2229] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:31:26.142657 6970 raft_consensus.cc:2258] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:26.143683 6970 ts_tablet_manager.cc:1905] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:31:26.145665 6970 ts_tablet_manager.cc:1918] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:31:26.145771 6970 log.cc:1196] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/7a0178d73ec140fb9ff9ecb688a04754
I20260809 09:31:26.146066 6970 ts_tablet_manager.cc:1939] T 7a0178d73ec140fb9ff9ecb688a04754 P 6e4cbc653e094b7982e01b80732bc74e: Deleting consensus metadata
I20260809 09:31:26.146922 6070 catalog_manager.cc:4841] TS 6e4cbc653e094b7982e01b80732bc74e (127.9.102.3:33771): tablet 7a0178d73ec140fb9ff9ecb688a04754 (table test-table [id=68ef3db1a8174f79b89016806789d25c]) successfully deleted
W20260809 09:31:26.148131 6067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:43911 (error 108)
I20260809 09:31:26.148667 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 6451
I20260809 09:31:26.149120 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.148836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:43911 (local address 127.0.0.1:42506)
0809 09:31:26.148986 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:26.149045 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:43911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260809 09:31:26.156937 6067 catalog_manager.cc:4584] TS 3fd72c20911a4c46a5bac4678d5e706a (127.9.102.1:43911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a0178d73ec140fb9ff9ecb688a04754: Network error: Client connection negotiation failed: client connection to 127.9.102.1:43911: connect: Connection refused (error 111)
I20260809 09:31:26.163061 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 6673
I20260809 09:31:26.169823 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 6043
2026-08-09T09:31:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2565 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T09:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:26Z Disabled control of system clock
I20260809 09:31:26.204396 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39451
--webserver_interface=127.9.102.62
--webserver_port=0
--builtin_ntp_servers=127.9.102.20:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 09:31:26.300729 6976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:26.300957 6976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:26.301017 6976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:26.304476 6976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:26.304551 6976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:26.304584 6976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:26.304618 6976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:26.304656 6976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:26.308954 6976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:26.309737 6976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:26.310822 6976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:26.317986 6988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:26.317973 6984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:26.317986 6985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:26.318203 6976 server_base.cc:1034] running on GCE node
I20260809 09:31:26.318697 6976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:26.319514 6976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:26.320667 6976 hybrid_clock.cc:648] HybridClock initialized: now 1786267886320639 us; error 38 us; skew 500 ppm
I20260809 09:31:26.322145 6976 webserver.cc:460] Webserver started at http://127.9.102.62:45879/ using document root <none> and password file <none>
I20260809 09:31:26.322572 6976 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:26.322624 6976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:26.322758 6976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:26.324455 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "27140a3bc2964823810fdb5a540ac6b6"
format_stamp: "Formatted at 2026-08-09 09:31:26 on dist-test-slave-7vjv"
server_key: "8b8aa33f11c6196cd077aba6323a35ba"
server_key_iv: "7f691f20b39949e3bd623b24547443ba"
server_key_version: "encryptionkey@0"
I20260809 09:31:26.324889 6976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "27140a3bc2964823810fdb5a540ac6b6"
format_stamp: "Formatted at 2026-08-09 09:31:26 on dist-test-slave-7vjv"
server_key: "8b8aa33f11c6196cd077aba6323a35ba"
server_key_iv: "7f691f20b39949e3bd623b24547443ba"
server_key_version: "encryptionkey@0"
I20260809 09:31:26.328073 6976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:31:26.330224 7003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:26.331338 6976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 09:31:26.331465 6976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "27140a3bc2964823810fdb5a540ac6b6"
format_stamp: "Formatted at 2026-08-09 09:31:26 on dist-test-slave-7vjv"
server_key: "8b8aa33f11c6196cd077aba6323a35ba"
server_key_iv: "7f691f20b39949e3bd623b24547443ba"
server_key_version: "encryptionkey@0"
I20260809 09:31:26.331560 6976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:26.343990 6976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:26.344537 6976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:26.344707 6976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:26.350894 6976 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:39451
I20260809 09:31:26.350931 7080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:39451 every 8 connection(s)
I20260809 09:31:26.351720 6976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:26.354422 7082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:26.359215 7082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6: Bootstrap starting.
I20260809 09:31:26.359540 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 6976
I20260809 09:31:26.359699 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:26.359994 9624 external_mini_cluster.cc:1428] Setting key a1a089153bec3346fa5d818c18101f90
I20260809 09:31:26.361852 7082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:26.362535 7082 log.cc:824] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:26.364494 7082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6: No bootstrap required, opened a new log
I20260809 09:31:26.368582 7082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27140a3bc2964823810fdb5a540ac6b6" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39451 } }
I20260809 09:31:26.368768 7082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:26.368834 7082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27140a3bc2964823810fdb5a540ac6b6, State: Initialized, Role: FOLLOWER
I20260809 09:31:26.368848 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.361308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34834 (local address 127.9.102.62:39451)
0809 09:31:26.361881 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:26.361916 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:26.361958 (+ 42us) server_negotiation.cc:407] Connection header received
0809 09:31:26.362517 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:26.362531 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:26.362965 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:26.363333 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:26.363635 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.364617 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.365804 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.366253 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.366367 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.367186 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:26.367191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:26.367203 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:26.367549 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:26.367605 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:26.367624 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:26.367697 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:26.367888 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 09:31:26.368182 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":123,"threads_started":1}
I20260809 09:31:26.369297 7082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27140a3bc2964823810fdb5a540ac6b6" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39451 } }
I20260809 09:31:26.369414 7082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:26.369499 7082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:26.369637 7082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:26.370411 7082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27140a3bc2964823810fdb5a540ac6b6" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39451 } }
I20260809 09:31:26.370726 7082 leader_election.cc:302] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27140a3bc2964823810fdb5a540ac6b6; no voters:
I20260809 09:31:26.370996 7082 leader_election.cc:288] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:26.371095 7091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:26.371325 7091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [term 1 LEADER]: Becoming Leader. State: Replica: 27140a3bc2964823810fdb5a540ac6b6, State: Running, Role: LEADER
I20260809 09:31:26.371592 7091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27140a3bc2964823810fdb5a540ac6b6" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39451 } }
I20260809 09:31:26.372048 7082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:26.373102 7097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27140a3bc2964823810fdb5a540ac6b6. Latest consensus state: current_term: 1 leader_uuid: "27140a3bc2964823810fdb5a540ac6b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27140a3bc2964823810fdb5a540ac6b6" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39451 } } }
I20260809 09:31:26.373214 7097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:26.373560 7116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:26.373543 7096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27140a3bc2964823810fdb5a540ac6b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27140a3bc2964823810fdb5a540ac6b6" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39451 } } }
I20260809 09:31:26.373653 7096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:26.376353 7116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:26.380872 7116 catalog_manager.cc:1347] Generated new cluster ID: d737f2e9eb4242118498a35d6fddc5ef
I20260809 09:31:26.380928 7116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:26.409554 7116 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:26.410627 7116 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:26.416174 7116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6: Generated new TSK 0
I20260809 09:31:26.416801 7116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:26.427395 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39451
--builtin_ntp_servers=127.9.102.20:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 09:31:26.527493 7136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:26.527697 7136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:26.527727 7136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:26.531131 7136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:26.531175 7136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:26.531235 7136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:31:26.536870 7136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:26.541664 7136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:26.543124 7136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:26.551422 7152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:26.551445 7157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:26.551472 7151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:26.551641 7136 server_base.cc:1034] running on GCE node
I20260809 09:31:26.552044 7136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:26.552568 7136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:26.553750 7136 hybrid_clock.cc:648] HybridClock initialized: now 1786267886553729 us; error 40 us; skew 500 ppm
I20260809 09:31:26.555202 7136 webserver.cc:460] Webserver started at http://127.9.102.1:41871/ using document root <none> and password file <none>
I20260809 09:31:26.555675 7136 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:26.555747 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:26.555959 7136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:26.557765 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "76d0716b63ee48fb8c09b9f92dc83bb0"
format_stamp: "Formatted at 2026-08-09 09:31:26 on dist-test-slave-7vjv"
server_key: "65cb79468e3ef76a4d9da340f5e709ce"
server_key_iv: "92008576072ffce0a56f948a417ddc7f"
server_key_version: "encryptionkey@0"
I20260809 09:31:26.558252 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "76d0716b63ee48fb8c09b9f92dc83bb0"
format_stamp: "Formatted at 2026-08-09 09:31:26 on dist-test-slave-7vjv"
server_key: "65cb79468e3ef76a4d9da340f5e709ce"
server_key_iv: "92008576072ffce0a56f948a417ddc7f"
server_key_version: "encryptionkey@0"
I20260809 09:31:26.561533 7136 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260809 09:31:26.563591 7169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:26.564734 7136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 09:31:26.564838 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "76d0716b63ee48fb8c09b9f92dc83bb0"
format_stamp: "Formatted at 2026-08-09 09:31:26 on dist-test-slave-7vjv"
server_key: "65cb79468e3ef76a4d9da340f5e709ce"
server_key_iv: "92008576072ffce0a56f948a417ddc7f"
server_key_version: "encryptionkey@0"
I20260809 09:31:26.564914 7136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:26.598101 7136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:26.598615 7136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:26.598760 7136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:26.599267 7136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:26.600171 7136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:26.600224 7136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:26.600265 7136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:26.600288 7136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:26.609377 7136 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:39621
I20260809 09:31:26.609398 7355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:39621 every 8 connection(s)
I20260809 09:31:26.610167 7136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:31:26.614250 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 7136
I20260809 09:31:26.614369 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:31:26.614732 9624 external_mini_cluster.cc:1428] Setting key 4fe1536ca414dd4067b7896adfcd23e4
I20260809 09:31:26.616756 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39451
--builtin_ntp_servers=127.9.102.20:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 09:31:26.619537 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.611853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:50169 (local address 127.9.102.62:39451)
0809 09:31:26.611992 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:26.611996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:26.613046 (+ 1050us) server_negotiation.cc:407] Connection header received
0809 09:31:26.614456 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:26.614461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:26.614526 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:26.614644 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:26.615497 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.616025 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.617563 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.617772 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.617856 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.619033 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:26.619036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:26.619040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:26.619056 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:26.619079 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:26.619090 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:26.619166 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:26.619299 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 09:31:26.619364 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 09:31:26.620266 7359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.612321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39451 (local address 127.9.102.1:50169)
0809 09:31:26.612909 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:26.612944 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:26.614209 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:26.614776 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:26.614800 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:26.614887 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:26.615370 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.615384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.616213 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.616216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.617448 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.617462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.617924 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.617929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.618127 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.618794 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:31:26.618795 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:26.618829 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:31:26.618885 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:26.619172 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:26.619179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:26.619183 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 09:31:26.619266 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 09:31:26.619455 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
I20260809 09:31:26.621352 7356 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39451
I20260809 09:31:26.621574 7356 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:26.622166 7356 heartbeater.cc:511] Master 127.9.102.62:39451 requested a full tablet report, sending...
I20260809 09:31:26.623633 7028 ts_manager.cc:194] Registered new tserver with Master: 76d0716b63ee48fb8c09b9f92dc83bb0 (127.9.102.1:39621)
I20260809 09:31:26.624770 7028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.102.1:50169
W20260809 09:31:26.716043 7360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:26.716238 7360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:26.716267 7360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:26.719664 7360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:26.719712 7360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:26.719758 7360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:31:26.723892 7360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:26.724694 7360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:26.725742 7360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:26.733758 7371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:26.733759 7368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:26.733837 7367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:26.733999 7360 server_base.cc:1034] running on GCE node
I20260809 09:31:26.734444 7360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:26.735045 7360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:26.736177 7360 hybrid_clock.cc:648] HybridClock initialized: now 1786267886736153 us; error 35 us; skew 500 ppm
I20260809 09:31:26.737682 7360 webserver.cc:460] Webserver started at http://127.9.102.2:43047/ using document root <none> and password file <none>
I20260809 09:31:26.738180 7360 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:26.738234 7360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:26.738467 7360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:26.740207 7360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "ad20f9add63742f3b19ef6b3f76656a4"
format_stamp: "Formatted at 2026-08-09 09:31:26 on dist-test-slave-7vjv"
server_key: "3f9051ffe1be9d55d6b79eaa7185aa3d"
server_key_iv: "dbfe880dbaecce8c605a29caa4870176"
server_key_version: "encryptionkey@0"
I20260809 09:31:26.740734 7360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "ad20f9add63742f3b19ef6b3f76656a4"
format_stamp: "Formatted at 2026-08-09 09:31:26 on dist-test-slave-7vjv"
server_key: "3f9051ffe1be9d55d6b79eaa7185aa3d"
server_key_iv: "dbfe880dbaecce8c605a29caa4870176"
server_key_version: "encryptionkey@0"
I20260809 09:31:26.743856 7360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:31:26.745816 7379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:26.746775 7360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:26.746881 7360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "ad20f9add63742f3b19ef6b3f76656a4"
format_stamp: "Formatted at 2026-08-09 09:31:26 on dist-test-slave-7vjv"
server_key: "3f9051ffe1be9d55d6b79eaa7185aa3d"
server_key_iv: "dbfe880dbaecce8c605a29caa4870176"
server_key_version: "encryptionkey@0"
I20260809 09:31:26.746968 7360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:26.754477 7360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:26.754956 7360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:26.755116 7360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:26.755607 7360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:26.756453 7360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:26.756511 7360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:26.756592 7360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:26.756624 7360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:26.764609 7360 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:38035
I20260809 09:31:26.764637 7578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:38035 every 8 connection(s)
I20260809 09:31:26.765477 7360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:31:26.773907 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.767071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:44203 (local address 127.9.102.62:39451)
0809 09:31:26.767194 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:26.767198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:26.768235 (+ 1037us) server_negotiation.cc:407] Connection header received
0809 09:31:26.769577 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:26.769580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:26.769637 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:26.769734 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:26.770516 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.771036 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.772136 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.772343 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.772451 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.773425 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:26.773429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:26.773431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:26.773444 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:26.773470 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:26.773477 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:26.773551 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:26.773699 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 09:31:26.773756 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 09:31:26.774067 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 7360
I20260809 09:31:26.774183 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:31:26.774451 9624 external_mini_cluster.cc:1428] Setting key 15ba7bd5cb94b77ffc9db4805baf8017
I20260809 09:31:26.774757 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.767408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39451 (local address 127.9.102.2:44203)
0809 09:31:26.768063 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:26.768135 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:26.769385 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:26.769885 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:26.769892 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:26.769962 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:26.770352 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.770364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.771164 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.771168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.771995 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.772004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.772482 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.772485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.772643 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.773174 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:31:26.773176 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:26.773207 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:31:26.773259 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:26.773554 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:26.773564 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:26.773571 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 09:31:26.773685 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 09:31:26.773948 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":121,"threads_started":1}
I20260809 09:31:26.775727 7579 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39451
I20260809 09:31:26.775938 7579 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:26.776336 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39451
--builtin_ntp_servers=127.9.102.20:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 09:31:26.776401 7579 heartbeater.cc:511] Master 127.9.102.62:39451 requested a full tablet report, sending...
I20260809 09:31:26.777305 7028 ts_manager.cc:194] Registered new tserver with Master: ad20f9add63742f3b19ef6b3f76656a4 (127.9.102.2:38035)
I20260809 09:31:26.777804 7028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.102.2:44203
W20260809 09:31:26.878451 7584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:26.878651 7584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:26.878683 7584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:26.882225 7584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:26.882265 7584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:26.882328 7584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:31:26.886642 7584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:26.887526 7584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:26.888751 7584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:26.897217 7595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:26.897236 7594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:26.897236 7601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:26.897454 7584 server_base.cc:1034] running on GCE node
I20260809 09:31:26.897944 7584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:26.898545 7584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:26.899766 7584 hybrid_clock.cc:648] HybridClock initialized: now 1786267886899738 us; error 55 us; skew 500 ppm
I20260809 09:31:26.901625 7584 webserver.cc:460] Webserver started at http://127.9.102.3:36283/ using document root <none> and password file <none>
I20260809 09:31:26.902205 7584 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:26.902302 7584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:26.902529 7584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:26.904412 7584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "27772fb7537641c3b936febbe1e4c7e7"
format_stamp: "Formatted at 2026-08-09 09:31:26 on dist-test-slave-7vjv"
server_key: "c9157556c31dc3aa2ffd22e5703e8f78"
server_key_iv: "55c16d2a20f9e0b594eeb29e2641e75f"
server_key_version: "encryptionkey@0"
I20260809 09:31:26.904973 7584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "27772fb7537641c3b936febbe1e4c7e7"
format_stamp: "Formatted at 2026-08-09 09:31:26 on dist-test-slave-7vjv"
server_key: "c9157556c31dc3aa2ffd22e5703e8f78"
server_key_iv: "55c16d2a20f9e0b594eeb29e2641e75f"
server_key_version: "encryptionkey@0"
I20260809 09:31:26.908432 7584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260809 09:31:26.910606 7612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:26.911636 7584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:26.911768 7584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "27772fb7537641c3b936febbe1e4c7e7"
format_stamp: "Formatted at 2026-08-09 09:31:26 on dist-test-slave-7vjv"
server_key: "c9157556c31dc3aa2ffd22e5703e8f78"
server_key_iv: "55c16d2a20f9e0b594eeb29e2641e75f"
server_key_version: "encryptionkey@0"
I20260809 09:31:26.911893 7584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:26.945326 7584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:26.945883 7584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:26.946071 7584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:26.946620 7584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:26.947491 7584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:26.947578 7584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:26.947652 7584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:26.947705 7584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:26.957199 7584 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:38401
I20260809 09:31:26.957245 7806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:38401 every 8 connection(s)
I20260809 09:31:26.958001 7584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:31:26.962889 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 7584
I20260809 09:31:26.962991 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:31:26.963228 9624 external_mini_cluster.cc:1428] Setting key e33f5f7ce937e98005d708cf5a14a552
I20260809 09:31:26.966770 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.959669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:49779 (local address 127.9.102.62:39451)
0809 09:31:26.959886 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:26.959890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:26.960824 (+ 934us) server_negotiation.cc:407] Connection header received
0809 09:31:26.962068 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:26.962071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:26.962126 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:26.962222 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:26.962953 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.963528 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.965088 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.965442 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.965526 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.966410 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:26.966412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:26.966414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:26.966424 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:26.966441 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:26.966448 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:26.966501 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:26.966633 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 09:31:26.966675 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260809 09:31:26.967455 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.959986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39451 (local address 127.9.102.3:49779)
0809 09:31:26.960690 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:26.960725 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:26.961872 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:26.962368 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:26.962374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:26.962457 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:26.962828 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.962838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.963697 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.963700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.964968 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.964979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.965584 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.965587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.965731 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.966200 (+ 469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:31:26.966202 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:26.966235 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:31:26.966287 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:26.966517 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:26.966529 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:26.966533 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 09:31:26.966615 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 09:31:26.966790 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":157,"threads_started":1}
I20260809 09:31:26.968482 7807 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39451
I20260809 09:31:26.968686 7807 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:26.969111 7807 heartbeater.cc:511] Master 127.9.102.62:39451 requested a full tablet report, sending...
I20260809 09:31:26.969974 7028 ts_manager.cc:194] Registered new tserver with Master: 27772fb7537641c3b936febbe1e4c7e7 (127.9.102.3:38401)
I20260809 09:31:26.970495 7028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.102.3:49779
I20260809 09:31:26.976830 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:31:26.981721 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.978632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.9.102.62:39451)
0809 09:31:26.978746 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:26.978749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:26.978934 (+ 185us) server_negotiation.cc:407] Connection header received
0809 09:31:26.979011 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:26.979013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:26.979051 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:26.979105 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:26.979319 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.979734 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.980505 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.980678 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.980746 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.981320 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:26.981322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:26.981324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:26.981351 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:26.981365 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:26.981371 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:26.981425 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:26.981585 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 09:31:26.981622 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 09:31:26.984050 7028 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:31:26.986758 7028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:31:26.999194 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.994206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:38035 (local address 127.0.0.1:43184)
0809 09:31:26.994524 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:26.994566 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:26.994676 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:26.995442 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:26.995447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:26.995469 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:26.995587 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.995593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.996885 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.996891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.997776 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.997785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.998739 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.998742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.998835 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.998869 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 09:31:26.998968 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 09:31:26.999065 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260809 09:31:26.999249 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.994338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.9.102.2:38035)
0809 09:31:26.994956 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:26.994966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:26.994993 (+ 27us) server_negotiation.cc:407] Connection header received
0809 09:31:26.995087 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:26.995092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:26.995280 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:26.995730 (+ 450us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:26.995860 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.996790 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.997895 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.998614 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.998735 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.998893 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:26.999003 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 09:31:26.999096 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":128,"threads_started":1}
I20260809 09:31:26.999683 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.994417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:39621 (local address 127.0.0.1:56640)
0809 09:31:26.994791 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:26.994805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:26.994907 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:26.996055 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:26.996058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:26.996076 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:26.996202 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.996209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.997436 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.997439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.998316 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.998324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.999244 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.999249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.999380 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.999405 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 09:31:26.999453 (+ 48us) client_negotiation.cc:240] Negotiation successful
0809 09:31:26.999536 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260809 09:31:27.000041 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.995189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:38401 (local address 127.0.0.1:33724)
0809 09:31:26.995629 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:26.995643 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:26.995721 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:26.996347 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:26.996349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:26.996363 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:26.996439 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.996443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.997542 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.997545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.998484 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:26.998494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.999575 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.999578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:26.999684 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.999715 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 09:31:26.999863 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 09:31:26.999927 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":21,"thread_start_us":99,"threads_started":1}
I20260809 09:31:27.000345 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.995433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.9.102.3:38401)
0809 09:31:26.995927 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:26.995937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:26.995957 (+ 20us) server_negotiation.cc:407] Connection header received
0809 09:31:26.996016 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:26.996021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:26.996203 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:26.996384 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:26.996546 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.997447 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.998618 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.999454 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.999560 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:26.999941 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:26.999994 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 09:31:27.000074 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":111,"threads_started":1}
I20260809 09:31:27.000387 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:26.994819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.9.102.1:39621)
0809 09:31:26.995595 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:26.995605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:26.995626 (+ 21us) server_negotiation.cc:407] Connection header received
0809 09:31:26.995701 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:26.995705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:26.995893 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:26.996068 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:26.996286 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.997319 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.998434 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:26.999122 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:26.999869 (+ 747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:27.000052 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:27.000141 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 09:31:27.000241 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":162,"threads_started":1}
I20260809 09:31:27.002413 7456 tablet_service.cc:1467] Processing CreateTablet for tablet d3e00140fa644f348129c9e73536e19f (DEFAULT_TABLE table=test-table [id=ca64f752b5824ba28d796b0f26e864b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:27.002681 7243 tablet_service.cc:1467] Processing CreateTablet for tablet d3e00140fa644f348129c9e73536e19f (DEFAULT_TABLE table=test-table [id=ca64f752b5824ba28d796b0f26e864b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:27.003489 7456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e00140fa644f348129c9e73536e19f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:27.003522 7243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e00140fa644f348129c9e73536e19f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:27.003566 7689 tablet_service.cc:1467] Processing CreateTablet for tablet d3e00140fa644f348129c9e73536e19f (DEFAULT_TABLE table=test-table [id=ca64f752b5824ba28d796b0f26e864b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:27.004525 7689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3e00140fa644f348129c9e73536e19f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:27.009123 7838 tablet_bootstrap.cc:492] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0: Bootstrap starting.
I20260809 09:31:27.010565 7841 tablet_bootstrap.cc:492] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4: Bootstrap starting.
I20260809 09:31:27.011153 7838 tablet_bootstrap.cc:654] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:27.011431 7844 tablet_bootstrap.cc:492] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7: Bootstrap starting.
I20260809 09:31:27.012056 7838 log.cc:824] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:27.012595 7841 tablet_bootstrap.cc:654] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:27.013259 7844 tablet_bootstrap.cc:654] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:27.013361 7841 log.cc:824] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:27.014025 7844 log.cc:824] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:27.014390 7838 tablet_bootstrap.cc:492] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0: No bootstrap required, opened a new log
I20260809 09:31:27.014592 7838 ts_tablet_manager.cc:1397] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 09:31:27.015146 7841 tablet_bootstrap.cc:492] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4: No bootstrap required, opened a new log
I20260809 09:31:27.015334 7841 ts_tablet_manager.cc:1397] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 09:31:27.015823 7844 tablet_bootstrap.cc:492] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7: No bootstrap required, opened a new log
I20260809 09:31:27.015976 7844 ts_tablet_manager.cc:1397] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 09:31:27.018186 7841 raft_consensus.cc:348] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad20f9add63742f3b19ef6b3f76656a4" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 38035 } } peers { permanent_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39621 } } peers { permanent_uuid: "27772fb7537641c3b936febbe1e4c7e7" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38401 } }
I20260809 09:31:27.018425 7841 raft_consensus.cc:374] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:27.018486 7841 raft_consensus.cc:729] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad20f9add63742f3b19ef6b3f76656a4, State: Initialized, Role: FOLLOWER
I20260809 09:31:27.018903 7841 consensus_queue.cc:260] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad20f9add63742f3b19ef6b3f76656a4" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 38035 } } peers { permanent_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39621 } } peers { permanent_uuid: "27772fb7537641c3b936febbe1e4c7e7" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38401 } }
I20260809 09:31:27.018859 7838 raft_consensus.cc:348] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad20f9add63742f3b19ef6b3f76656a4" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 38035 } } peers { permanent_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39621 } } peers { permanent_uuid: "27772fb7537641c3b936febbe1e4c7e7" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38401 } }
I20260809 09:31:27.019071 7838 raft_consensus.cc:374] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:27.019120 7838 raft_consensus.cc:729] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d0716b63ee48fb8c09b9f92dc83bb0, State: Initialized, Role: FOLLOWER
I20260809 09:31:27.019510 7838 consensus_queue.cc:260] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad20f9add63742f3b19ef6b3f76656a4" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 38035 } } peers { permanent_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39621 } } peers { permanent_uuid: "27772fb7537641c3b936febbe1e4c7e7" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38401 } }
I20260809 09:31:27.019760 7841 ts_tablet_manager.cc:1428] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260809 09:31:27.019773 7844 raft_consensus.cc:348] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad20f9add63742f3b19ef6b3f76656a4" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 38035 } } peers { permanent_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39621 } } peers { permanent_uuid: "27772fb7537641c3b936febbe1e4c7e7" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38401 } }
I20260809 09:31:27.020004 7844 raft_consensus.cc:374] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:27.020048 7844 raft_consensus.cc:729] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27772fb7537641c3b936febbe1e4c7e7, State: Initialized, Role: FOLLOWER
I20260809 09:31:27.020149 7579 heartbeater.cc:503] Master 127.9.102.62:39451 was elected leader, sending a full tablet report...
I20260809 09:31:27.020447 7838 ts_tablet_manager.cc:1428] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260809 09:31:27.020511 7844 consensus_queue.cc:260] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad20f9add63742f3b19ef6b3f76656a4" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 38035 } } peers { permanent_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39621 } } peers { permanent_uuid: "27772fb7537641c3b936febbe1e4c7e7" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38401 } }
I20260809 09:31:27.021075 7807 heartbeater.cc:503] Master 127.9.102.62:39451 was elected leader, sending a full tablet report...
I20260809 09:31:27.021327 7844 ts_tablet_manager.cc:1428] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260809 09:31:27.021929 7356 heartbeater.cc:503] Master 127.9.102.62:39451 was elected leader, sending a full tablet report...
I20260809 09:31:27.075349 7848 raft_consensus.cc:482] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:31:27.075515 7848 raft_consensus.cc:504] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad20f9add63742f3b19ef6b3f76656a4" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 38035 } } peers { permanent_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39621 } } peers { permanent_uuid: "27772fb7537641c3b936febbe1e4c7e7" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38401 } }
I20260809 09:31:27.076468 7848 leader_election.cc:288] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d0716b63ee48fb8c09b9f92dc83bb0 (127.9.102.1:39621), 27772fb7537641c3b936febbe1e4c7e7 (127.9.102.3:38401)
I20260809 09:31:27.079939 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:27.076701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:33625 (local address 127.9.102.1:39621)
0809 09:31:27.076828 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:27.076832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:27.076848 (+ 16us) server_negotiation.cc:407] Connection header received
0809 09:31:27.076923 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:27.076926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:27.076971 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:27.077041 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:27.077362 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:27.078102 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:27.079128 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:27.079456 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:27.079564 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:27.079631 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:27.079786 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 09:31:27.079846 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 09:31:27.079941 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:27.076627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:39621 (local address 127.9.102.2:33625)
0809 09:31:27.076750 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:27.076765 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:27.076841 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:27.077108 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:27.077113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:27.077135 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:27.077235 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:27.077240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:27.078215 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:27.078221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:27.079008 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:27.079017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:27.079559 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:27.079562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:27.079663 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:27.079689 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 09:31:27.079762 (+ 73us) client_negotiation.cc:240] Negotiation successful
0809 09:31:27.079826 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":3}
I20260809 09:31:27.080047 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:27.076783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:36977 (local address 127.9.102.3:38401)
0809 09:31:27.076887 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:27.076891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:27.077080 (+ 189us) server_negotiation.cc:407] Connection header received
0809 09:31:27.077237 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:27.077240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:27.077325 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:27.077399 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:27.077579 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:27.078237 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:27.079191 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:27.079531 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:27.079651 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:27.079703 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:27.079883 (+ 180us) server_negotiation.cc:299] Negotiation successful
0809 09:31:27.079939 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 09:31:27.080081 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:27.076657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:38401 (local address 127.9.102.2:36977)
0809 09:31:27.076997 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:27.077011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:27.077092 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:27.077404 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:27.077406 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:27.077418 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:27.077487 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:27.077490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:27.078361 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:27.078364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:27.079078 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:27.079085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:27.079676 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:27.079680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:27.079769 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:27.079785 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 09:31:27.079862 (+ 77us) client_negotiation.cc:240] Negotiation successful
0809 09:31:27.079921 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260809 09:31:27.080590 7721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e00140fa644f348129c9e73536e19f" candidate_uuid: "ad20f9add63742f3b19ef6b3f76656a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27772fb7537641c3b936febbe1e4c7e7" is_pre_election: true
I20260809 09:31:27.080832 7721 raft_consensus.cc:2454] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad20f9add63742f3b19ef6b3f76656a4 in term 0.
I20260809 09:31:27.081220 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e00140fa644f348129c9e73536e19f" candidate_uuid: "ad20f9add63742f3b19ef6b3f76656a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0" is_pre_election: true
I20260809 09:31:27.081243 7382 leader_election.cc:302] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27772fb7537641c3b936febbe1e4c7e7, ad20f9add63742f3b19ef6b3f76656a4; no voters:
I20260809 09:31:27.081453 7272 raft_consensus.cc:2454] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad20f9add63742f3b19ef6b3f76656a4 in term 0.
I20260809 09:31:27.081472 7848 raft_consensus.cc:2781] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:31:27.081537 7848 raft_consensus.cc:482] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:31:27.081604 7848 raft_consensus.cc:3037] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:27.082515 7848 raft_consensus.cc:504] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad20f9add63742f3b19ef6b3f76656a4" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 38035 } } peers { permanent_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39621 } } peers { permanent_uuid: "27772fb7537641c3b936febbe1e4c7e7" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38401 } }
I20260809 09:31:27.082782 7848 leader_election.cc:288] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [CANDIDATE]: Term 1 election: Requested vote from peers 76d0716b63ee48fb8c09b9f92dc83bb0 (127.9.102.1:39621), 27772fb7537641c3b936febbe1e4c7e7 (127.9.102.3:38401)
I20260809 09:31:27.083063 7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e00140fa644f348129c9e73536e19f" candidate_uuid: "ad20f9add63742f3b19ef6b3f76656a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0"
I20260809 09:31:27.083148 7272 raft_consensus.cc:3037] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:27.084270 7272 raft_consensus.cc:2454] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad20f9add63742f3b19ef6b3f76656a4 in term 1.
I20260809 09:31:27.084265 7721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3e00140fa644f348129c9e73536e19f" candidate_uuid: "ad20f9add63742f3b19ef6b3f76656a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27772fb7537641c3b936febbe1e4c7e7"
I20260809 09:31:27.084412 7721 raft_consensus.cc:3037] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:27.084661 7384 leader_election.cc:302] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d0716b63ee48fb8c09b9f92dc83bb0, ad20f9add63742f3b19ef6b3f76656a4; no voters:
I20260809 09:31:27.084820 7848 raft_consensus.cc:2781] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:27.085090 7848 raft_consensus.cc:686] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 1 LEADER]: Becoming Leader. State: Replica: ad20f9add63742f3b19ef6b3f76656a4, State: Running, Role: LEADER
I20260809 09:31:27.085388 7848 consensus_queue.cc:237] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad20f9add63742f3b19ef6b3f76656a4" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 38035 } } peers { permanent_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39621 } } peers { permanent_uuid: "27772fb7537641c3b936febbe1e4c7e7" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38401 } }
I20260809 09:31:27.085618 7721 raft_consensus.cc:2454] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad20f9add63742f3b19ef6b3f76656a4 in term 1.
I20260809 09:31:27.088228 7028 catalog_manager.cc:5495] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad20f9add63742f3b19ef6b3f76656a4 (127.9.102.2). New cstate: current_term: 1 leader_uuid: "ad20f9add63742f3b19ef6b3f76656a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad20f9add63742f3b19ef6b3f76656a4" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 38035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27772fb7537641c3b936febbe1e4c7e7" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38401 } health_report { overall_health: UNKNOWN } } }
I20260809 09:31:27.097911 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:27.094988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.9.102.2:38035)
0809 09:31:27.095089 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:27.095092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:27.095102 (+ 10us) server_negotiation.cc:407] Connection header received
0809 09:31:27.095187 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:27.095188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:27.095226 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:27.095294 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:31:27.095509 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:27.095902 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:27.096730 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:27.096916 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:27.097014 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:27.097314 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:31:27.097635 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:31:27.097713 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:27.097787 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 09:31:27.097822 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260809 09:31:27.102774 7272 raft_consensus.cc:1261] T d3e00140fa644f348129c9e73536e19f P 76d0716b63ee48fb8c09b9f92dc83bb0 [term 1 FOLLOWER]: Refusing update from remote peer ad20f9add63742f3b19ef6b3f76656a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:31:27.102823 7721 raft_consensus.cc:1261] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7 [term 1 FOLLOWER]: Refusing update from remote peer ad20f9add63742f3b19ef6b3f76656a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:31:27.103420 7852 consensus_queue.cc:1035] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d0716b63ee48fb8c09b9f92dc83bb0" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:31:27.103605 7848 consensus_queue.cc:1035] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27772fb7537641c3b936febbe1e4c7e7" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:31:27.109390 7856 mvcc.cc:205] Tried to move back new op lower bound from 7316553265567129600 to 7316553265504968704. Current Snapshot: MvccSnapshot[applied={T|T < 7316553265567129600}]
I20260809 09:31:27.110580 7858 mvcc.cc:205] Tried to move back new op lower bound from 7316553265567129600 to 7316553265504968704. Current Snapshot: MvccSnapshot[applied={T|T < 7316553265567129600}]
I20260809 09:31:27.111213 7859 mvcc.cc:205] Tried to move back new op lower bound from 7316553265567129600 to 7316553265504968704. Current Snapshot: MvccSnapshot[applied={T|T < 7316553265567129600}]
I20260809 09:31:27.117007 7027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:31:27.117148 7027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:31:27.119240 7027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27140a3bc2964823810fdb5a540ac6b6: Sending DeleteTablet for 3 replicas of tablet d3e00140fa644f348129c9e73536e19f
I20260809 09:31:27.120044 7456 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e00140fa644f348129c9e73536e19f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:27 UTC) from {username='slave'} at 127.0.0.1:43184
I20260809 09:31:27.120077 7689 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e00140fa644f348129c9e73536e19f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:27 UTC) from {username='slave'} at 127.0.0.1:33724
I20260809 09:31:27.120538 7874 tablet_replica.cc:330] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7: stopping tablet replica
I20260809 09:31:27.120556 7243 tablet_service.cc:1514] Processing DeleteTablet for tablet d3e00140fa644f348129c9e73536e19f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:27 UTC) from {username='slave'} at 127.0.0.1:56640
I20260809 09:31:27.120824 7874 raft_consensus.cc:2229] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:31:27.120889 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 7136
I20260809 09:31:27.121080 7874 raft_consensus.cc:2258] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:27.128628 7873 tablet_replica.cc:330] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4: stopping tablet replica
I20260809 09:31:27.128897 7873 raft_consensus.cc:2229] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:31:27.129181 7873 raft_consensus.cc:2258] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:27.130734 7873 ts_tablet_manager.cc:1905] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:31:27.133459 7873 ts_tablet_manager.cc:1918] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:31:27.133579 7873 log.cc:1196] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/d3e00140fa644f348129c9e73536e19f
I20260809 09:31:27.133975 7873 ts_tablet_manager.cc:1939] T d3e00140fa644f348129c9e73536e19f P ad20f9add63742f3b19ef6b3f76656a4: Deleting consensus metadata
I20260809 09:31:27.135063 7005 catalog_manager.cc:4841] TS ad20f9add63742f3b19ef6b3f76656a4 (127.9.102.2:38035): tablet d3e00140fa644f348129c9e73536e19f (table test-table [id=ca64f752b5824ba28d796b0f26e864b1]) successfully deleted
I20260809 09:31:27.136193 7874 ts_tablet_manager.cc:1905] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:31:27.138060 7874 ts_tablet_manager.cc:1918] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:31:27.138198 7874 log.cc:1196] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/d3e00140fa644f348129c9e73536e19f
I20260809 09:31:27.138505 7874 ts_tablet_manager.cc:1939] T d3e00140fa644f348129c9e73536e19f P 27772fb7537641c3b936febbe1e4c7e7: Deleting consensus metadata
I20260809 09:31:27.139357 7005 catalog_manager.cc:4841] TS 27772fb7537641c3b936febbe1e4c7e7 (127.9.102.3:38401): tablet d3e00140fa644f348129c9e73536e19f (table test-table [id=ca64f752b5824ba28d796b0f26e864b1]) successfully deleted
W20260809 09:31:27.140223 7007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:39621 (error 108)
I20260809 09:31:27.141057 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 7360
I20260809 09:31:27.141191 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:27.140894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:39621 (local address 127.0.0.1:56656)
0809 09:31:27.141013 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:27.141090 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:39621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 09:31:27.147993 7007 catalog_manager.cc:4584] TS 76d0716b63ee48fb8c09b9f92dc83bb0 (127.9.102.1:39621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3e00140fa644f348129c9e73536e19f: Network error: Client connection negotiation failed: client connection to 127.9.102.1:39621: connect: Connection refused (error 111)
I20260809 09:31:27.152603 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 7584
I20260809 09:31:27.161962 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 6976
2026-08-09T09:31:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (989 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:27 dist-test-slave-7vjv krb5kdc[7884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:27 dist-test-slave-7vjv krb5kdc[7884](info): set up 2 sockets
Aug 09 09:31:27 dist-test-slave-7vjv krb5kdc[7884](info): commencing operation
krb5kdc: starting...
W20260809 09:31:28.326359 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:28 dist-test-slave-7vjv krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:31:28.438491 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39447
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:28.539672 7904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:28.539882 7904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:28.539916 7904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:28.543573 7904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:28.543622 7904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:28.543641 7904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:28.543658 7904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:28.543715 7904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:28.548310 7904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:28.549151 7904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:28.549996 7904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:28.557372 7911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:28.557453 7916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:28.557351 7913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:28.557643 7904 server_base.cc:1034] running on GCE node
I20260809 09:31:28.558148 7904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:28.559001 7904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:28.560173 7904 hybrid_clock.cc:648] HybridClock initialized: now 1786267888560153 us; error 29 us; skew 500 ppm
Aug 09 09:31:28 dist-test-slave-7vjv krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267888, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:28.565944 7904 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:31:28.566670 7904 webserver.cc:460] Webserver started at http://127.9.102.62:37485/ using document root <none> and password file <none>
I20260809 09:31:28.567168 7904 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:28.567255 7904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:28.567461 7904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:28.569242 7904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "70619fe3c9ed471bb7d3d7fde46c043a"
format_stamp: "Formatted at 2026-08-09 09:31:28 on dist-test-slave-7vjv"
server_key: "20d90a6f199263313f900bb6f58163e5"
server_key_iv: "99a7407401281a2d8b9e359cdfa4b5fe"
server_key_version: "encryptionkey@0"
I20260809 09:31:28.569747 7904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "70619fe3c9ed471bb7d3d7fde46c043a"
format_stamp: "Formatted at 2026-08-09 09:31:28 on dist-test-slave-7vjv"
server_key: "20d90a6f199263313f900bb6f58163e5"
server_key_iv: "99a7407401281a2d8b9e359cdfa4b5fe"
server_key_version: "encryptionkey@0"
I20260809 09:31:28.572906 7904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:31:28.575065 7928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:28.576231 7904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 09:31:28.576359 7904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "70619fe3c9ed471bb7d3d7fde46c043a"
format_stamp: "Formatted at 2026-08-09 09:31:28 on dist-test-slave-7vjv"
server_key: "20d90a6f199263313f900bb6f58163e5"
server_key_iv: "99a7407401281a2d8b9e359cdfa4b5fe"
server_key_version: "encryptionkey@0"
I20260809 09:31:28.576459 7904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:28.588176 7904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:28.591163 7904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:28.591344 7904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:28.597834 7904 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:39447
I20260809 09:31:28.597847 8029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:39447 every 8 connection(s)
I20260809 09:31:28.598640 7904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:28.601497 8030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:28.604394 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 7904
I20260809 09:31:28.604547 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:28.604857 9624 external_mini_cluster.cc:1428] Setting key 0af3204533b8491b15ba219cdfab49cf
I20260809 09:31:28.607426 8030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a: Bootstrap starting.
I20260809 09:31:28.609899 8030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:28.610661 8030 log.cc:824] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:28.612375 8030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a: No bootstrap required, opened a new log
Aug 09 09:31:28 dist-test-slave-7vjv krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:28.615923 8030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70619fe3c9ed471bb7d3d7fde46c043a" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39447 } }
I20260809 09:31:28.616250 8030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:28.616344 8030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70619fe3c9ed471bb7d3d7fde46c043a, State: Initialized, Role: FOLLOWER
I20260809 09:31:28.616817 8030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70619fe3c9ed471bb7d3d7fde46c043a" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39447 } }
I20260809 09:31:28.616938 8030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:28.617017 8030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:28.617138 8030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:28.618068 8030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70619fe3c9ed471bb7d3d7fde46c043a" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39447 } }
I20260809 09:31:28.618405 8030 leader_election.cc:302] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70619fe3c9ed471bb7d3d7fde46c043a; no voters:
I20260809 09:31:28.618676 8030 leader_election.cc:288] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:28.618695 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:28.606208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.9.102.62:39447)
0809 09:31:28.606612 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:28.606633 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:28.606674 (+ 41us) server_negotiation.cc:407] Connection header received
0809 09:31:28.607462 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:28.607484 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:28.607860 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:28.608207 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:28.608834 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:28.610234 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:28.611057 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:28.611460 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:28.611622 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:28.614134 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:28.614139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:28.614149 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:28.614172 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:28.615743 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:28.616267 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:28.616274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:28.616279 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:28.616448 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:28.616730 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:28.616732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:28.616733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:28.617165 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:28.617381 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:28.617647 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 09:31:28.617919 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260809 09:31:28.618779 8037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:28.618956 8037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [term 1 LEADER]: Becoming Leader. State: Replica: 70619fe3c9ed471bb7d3d7fde46c043a, State: Running, Role: LEADER
I20260809 09:31:28.619225 8037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70619fe3c9ed471bb7d3d7fde46c043a" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39447 } }
I20260809 09:31:28.619843 8030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:28.620302 8039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70619fe3c9ed471bb7d3d7fde46c043a. Latest consensus state: current_term: 1 leader_uuid: "70619fe3c9ed471bb7d3d7fde46c043a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70619fe3c9ed471bb7d3d7fde46c043a" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39447 } } }
I20260809 09:31:28.620415 8039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:28.621133 8038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70619fe3c9ed471bb7d3d7fde46c043a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70619fe3c9ed471bb7d3d7fde46c043a" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39447 } } }
I20260809 09:31:28.621232 8038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a [sys.catalog]: This master's current role is: LEADER
W20260809 09:31:28.624343 8066 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:31:28.624423 8066 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:31:28.624480 8046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:28.625314 8046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:28.629993 8046 catalog_manager.cc:1347] Generated new cluster ID: 24572cbae1b848e19385fbd7a41d40cf
I20260809 09:31:28.630053 8046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:28.648793 8046 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:28.649921 8046 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:28.654836 8046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a: Generated new TSK 0
I20260809 09:31:28.655536 8046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:31:28.683123 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39447
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:28.785132 8075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:28.785351 8075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:28.785386 8075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:28.789090 8075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:28.789136 8075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:28.789247 8075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:31:28.793954 8075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:28.794909 8075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:28.795864 8075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:28.804759 8083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:28.804812 8086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:28.804812 8082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:28.805099 8075 server_base.cc:1034] running on GCE node
I20260809 09:31:28.805485 8075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:28.806123 8075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:28.807277 8075 hybrid_clock.cc:648] HybridClock initialized: now 1786267888807254 us; error 33 us; skew 500 ppm
Aug 09 09:31:28 dist-test-slave-7vjv krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267888, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:28.809870 8075 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:31:28.810720 8075 webserver.cc:460] Webserver started at http://127.9.102.1:45035/ using document root <none> and password file <none>
I20260809 09:31:28.811280 8075 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:28.811364 8075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:28.811589 8075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:28.813495 8075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "ce6499b4680a41f89bb50597aa705bda"
format_stamp: "Formatted at 2026-08-09 09:31:28 on dist-test-slave-7vjv"
server_key: "41a3b3922aede11f62d5d671ee108b6e"
server_key_iv: "c13fde803d3c60dd20a3869b0e48b5e6"
server_key_version: "encryptionkey@0"
I20260809 09:31:28.814031 8075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "ce6499b4680a41f89bb50597aa705bda"
format_stamp: "Formatted at 2026-08-09 09:31:28 on dist-test-slave-7vjv"
server_key: "41a3b3922aede11f62d5d671ee108b6e"
server_key_iv: "c13fde803d3c60dd20a3869b0e48b5e6"
server_key_version: "encryptionkey@0"
I20260809 09:31:28.817363 8075 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:31:28.819495 8098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:28.820806 8075 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:28.820966 8075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "ce6499b4680a41f89bb50597aa705bda"
format_stamp: "Formatted at 2026-08-09 09:31:28 on dist-test-slave-7vjv"
server_key: "41a3b3922aede11f62d5d671ee108b6e"
server_key_iv: "c13fde803d3c60dd20a3869b0e48b5e6"
server_key_version: "encryptionkey@0"
I20260809 09:31:28.821066 8075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:28.846727 8075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:28.849293 8075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:28.849478 8075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:28.849980 8075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:28.850798 8075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:28.850871 8075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:28.850940 8075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:28.850994 8075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:28.859223 8075 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:33845
I20260809 09:31:28.859299 8305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:33845 every 8 connection(s)
I20260809 09:31:28.860042 8075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
Aug 09 09:31:28 dist-test-slave-7vjv krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267888, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:28.869349 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 8075
I20260809 09:31:28.869493 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:31:28.869865 9624 external_mini_cluster.cc:1428] Setting key 6b8999b800c7cb3548fffc5bc43aa144
I20260809 09:31:28.872362 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:28.861606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:54331 (local address 127.9.102.62:39447)
0809 09:31:28.861740 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:28.861744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:28.862495 (+ 751us) server_negotiation.cc:407] Connection header received
0809 09:31:28.863620 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:28.863623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:28.863665 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:28.863733 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:28.864854 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:28.865353 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:28.866337 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:28.866538 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:28.866609 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:28.868863 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:28.868866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:28.868868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:28.868889 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:28.870441 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:28.870927 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:28.870930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:28.870932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:28.871015 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:28.871339 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:28.871342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:28.871344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:28.871510 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:28.871613 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:28.872048 (+ 435us) server_negotiation.cc:299] Negotiation successful
0809 09:31:28.872202 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 09:31:28.873018 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:28.861979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39447 (local address 127.9.102.1:54331)
0809 09:31:28.862364 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:28.862399 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:28.863446 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:28.863863 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:28.863868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:28.864265 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:28.864726 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:28.864738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:28.865453 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:28.865455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:28.866240 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:28.866248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:28.866653 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:28.866655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:28.866773 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:28.867308 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:31:28.867324 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:28.868726 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:28.870564 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:28.870569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:28.870581 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:28.870819 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:28.871110 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:28.871113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:28.871115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:28.871243 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:28.871613 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:28.871619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:28.871827 (+ 208us) client_negotiation.cc:769] Sending connection context
0809 09:31:28.871998 (+ 171us) client_negotiation.cc:240] Negotiation successful
0809 09:31:28.872324 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
I20260809 09:31:28.874264 8306 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39447
I20260809 09:31:28.874496 8306 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:28.875015 8306 heartbeater.cc:511] Master 127.9.102.62:39447 requested a full tablet report, sending...
I20260809 09:31:28.876601 7961 ts_manager.cc:194] Registered new tserver with Master: ce6499b4680a41f89bb50597aa705bda (127.9.102.1:33845)
I20260809 09:31:28.877663 7961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:54331
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:31:28.897256 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39447
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:28.996054 8313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:28.996366 8313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:28.996433 8313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:28.999821 8313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:28.999897 8313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:28.999980 8313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:31:29.004626 8313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:29.005493 8313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:29.006350 8313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:29.014971 8324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:29.014972 8319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:29.014986 8320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:29.015329 8313 server_base.cc:1034] running on GCE node
I20260809 09:31:29.015712 8313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:29.016314 8313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:29.017485 8313 hybrid_clock.cc:648] HybridClock initialized: now 1786267889017458 us; error 39 us; skew 500 ppm
Aug 09 09:31:29 dist-test-slave-7vjv krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267889, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:29.019981 8313 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:31:29.020861 8313 webserver.cc:460] Webserver started at http://127.9.102.2:33529/ using document root <none> and password file <none>
I20260809 09:31:29.021355 8313 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:29.021438 8313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:29.021643 8313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:29.023355 8313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "b6c673fea27940e08cd7e0c8477c6b00"
format_stamp: "Formatted at 2026-08-09 09:31:29 on dist-test-slave-7vjv"
server_key: "ebc2a37005d8bd7b3275749328cc7562"
server_key_iv: "9770692832602c69d2dee51e9065043c"
server_key_version: "encryptionkey@0"
I20260809 09:31:29.023931 8313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "b6c673fea27940e08cd7e0c8477c6b00"
format_stamp: "Formatted at 2026-08-09 09:31:29 on dist-test-slave-7vjv"
server_key: "ebc2a37005d8bd7b3275749328cc7562"
server_key_iv: "9770692832602c69d2dee51e9065043c"
server_key_version: "encryptionkey@0"
I20260809 09:31:29.027395 8313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 09:31:29.029492 8338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:29.030583 8313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:31:29.030733 8313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "b6c673fea27940e08cd7e0c8477c6b00"
format_stamp: "Formatted at 2026-08-09 09:31:29 on dist-test-slave-7vjv"
server_key: "ebc2a37005d8bd7b3275749328cc7562"
server_key_iv: "9770692832602c69d2dee51e9065043c"
server_key_version: "encryptionkey@0"
I20260809 09:31:29.030836 8313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:29.040424 8313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:29.042822 8313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:29.043016 8313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:29.043516 8313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:29.044334 8313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:29.044415 8313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:29.044477 8313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:29.044550 8313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:29.052589 8313 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:34281
I20260809 09:31:29.052650 8530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:34281 every 8 connection(s)
I20260809 09:31:29.053402 8313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
Aug 09 09:31:29 dist-test-slave-7vjv krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267889, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:29.063169 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 8313
I20260809 09:31:29.063299 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:31:29.063601 9624 external_mini_cluster.cc:1428] Setting key c1e8895a2ff29751185f5eb902e65f48
I20260809 09:31:29.066277 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.055017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:59193 (local address 127.9.102.62:39447)
0809 09:31:29.055149 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:29.055153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:29.055897 (+ 744us) server_negotiation.cc:407] Connection header received
0809 09:31:29.057203 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:29.057206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:29.057261 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:29.057332 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:29.058326 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.058822 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.059912 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.060189 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.060290 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.062460 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:29.062463 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:29.062466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:29.062492 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:29.064007 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:29.064758 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:29.064762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:29.064764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:29.064875 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:29.065250 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:29.065252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:29.065253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:29.065409 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:29.065481 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:29.066043 (+ 562us) server_negotiation.cc:299] Negotiation successful
0809 09:31:29.066145 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:31:29.067267 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.055339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39447 (local address 127.9.102.2:59193)
0809 09:31:29.055761 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:29.055801 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:29.057008 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:29.057482 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:29.057488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:29.057795 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:29.058189 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.058201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.058949 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.058964 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.059780 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.059789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.060335 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.060339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.060490 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.061006 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:31:29.061021 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:29.062309 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:29.064188 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:29.064193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:29.064204 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:29.064600 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:29.064967 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:29.064969 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:29.064970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:29.065155 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:29.065513 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:29.065528 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:29.065795 (+ 267us) client_negotiation.cc:769] Sending connection context
0809 09:31:29.065988 (+ 193us) client_negotiation.cc:240] Negotiation successful
0809 09:31:29.066346 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":103,"threads_started":1}
I20260809 09:31:29.068307 8531 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39447
I20260809 09:31:29.068570 8531 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:29.069065 8531 heartbeater.cc:511] Master 127.9.102.62:39447 requested a full tablet report, sending...
I20260809 09:31:29.070119 7961 ts_manager.cc:194] Registered new tserver with Master: b6c673fea27940e08cd7e0c8477c6b00 (127.9.102.2:34281)
I20260809 09:31:29.070717 7961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:59193
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:31:29.090628 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39447
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:29.188432 8537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:29.188651 8537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:29.188711 8537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:29.192195 8537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:29.192263 8537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:29.192345 8537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:31:29.196863 8537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:39447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:29.197732 8537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:29.198542 8537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:29.206912 8545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:29.206948 8547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:29.206959 8543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:29.207136 8537 server_base.cc:1034] running on GCE node
I20260809 09:31:29.207561 8537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:29.208163 8537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:29.209350 8537 hybrid_clock.cc:648] HybridClock initialized: now 1786267889209333 us; error 33 us; skew 500 ppm
Aug 09 09:31:29 dist-test-slave-7vjv krb5kdc[7884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267889, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:29.212015 8537 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:31:29.212766 8537 webserver.cc:460] Webserver started at http://127.9.102.3:34251/ using document root <none> and password file <none>
I20260809 09:31:29.213224 8537 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:29.213276 8537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:29.213413 8537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:29.215076 8537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "1952653c894c44b4af6fb52ef96c3ba6"
format_stamp: "Formatted at 2026-08-09 09:31:29 on dist-test-slave-7vjv"
server_key: "4ba18afacd77f05f3d4c945df023014f"
server_key_iv: "6c3a788737cdb679f5a0d7190af553f6"
server_key_version: "encryptionkey@0"
I20260809 09:31:29.215526 8537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "1952653c894c44b4af6fb52ef96c3ba6"
format_stamp: "Formatted at 2026-08-09 09:31:29 on dist-test-slave-7vjv"
server_key: "4ba18afacd77f05f3d4c945df023014f"
server_key_iv: "6c3a788737cdb679f5a0d7190af553f6"
server_key_version: "encryptionkey@0"
I20260809 09:31:29.218637 8537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:31:29.220727 8561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:29.221856 8537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:29.222009 8537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "1952653c894c44b4af6fb52ef96c3ba6"
format_stamp: "Formatted at 2026-08-09 09:31:29 on dist-test-slave-7vjv"
server_key: "4ba18afacd77f05f3d4c945df023014f"
server_key_iv: "6c3a788737cdb679f5a0d7190af553f6"
server_key_version: "encryptionkey@0"
I20260809 09:31:29.222088 8537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:29.232596 8537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:29.234900 8537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:29.235054 8537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:29.235538 8537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:29.236303 8537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:29.236346 8537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:29.236390 8537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:29.236409 8537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:29.244156 8769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:38141 every 8 connection(s)
I20260809 09:31:29.244185 8537 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:38141
I20260809 09:31:29.244978 8537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:31:29.245713 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 8537
I20260809 09:31:29.245814 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:31:29.246094 9624 external_mini_cluster.cc:1428] Setting key 618ba0d0e75dda751766be77da092b65
Aug 09 09:31:29 dist-test-slave-7vjv krb5kdc[7884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267889, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:29.257377 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.246773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:37911 (local address 127.9.102.62:39447)
0809 09:31:29.246912 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:29.246917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:29.247790 (+ 873us) server_negotiation.cc:407] Connection header received
0809 09:31:29.249067 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:29.249070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:29.249119 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:29.249186 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:29.250181 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.250703 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.251773 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.252008 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.252131 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.254535 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:29.254537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:29.254539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:29.254561 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:29.255694 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:29.256156 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:29.256162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:29.256163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:29.256232 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:29.256517 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:29.256519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:29.256520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:29.256671 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:29.256741 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:29.257157 (+ 416us) server_negotiation.cc:299] Negotiation successful
0809 09:31:29.257268 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 09:31:29.258140 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.247144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39447 (local address 127.9.102.3:37911)
0809 09:31:29.247658 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:29.247697 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:29.248877 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:29.249327 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:29.249333 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:29.249620 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:29.250042 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.250054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.250847 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.250850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.251643 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.251653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.252168 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.252172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.252323 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.252994 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:31:29.253008 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:29.254396 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:29.255788 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:29.255792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:29.255801 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:29.256020 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:29.256306 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:29.256308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:29.256309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:29.256425 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:29.256754 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:29.256759 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:29.256941 (+ 182us) client_negotiation.cc:769] Sending connection context
0809 09:31:29.257083 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 09:31:29.257341 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":139,"threads_started":1}
I20260809 09:31:29.259244 8771 heartbeater.cc:347] Connected to a master server at 127.9.102.62:39447
I20260809 09:31:29.259452 8771 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:29.259933 8771 heartbeater.cc:511] Master 127.9.102.62:39447 requested a full tablet report, sending...
I20260809 09:31:29.260860 7961 ts_manager.cc:194] Registered new tserver with Master: 1952653c894c44b4af6fb52ef96c3ba6 (127.9.102.3:38141)
I20260809 09:31:29.261332 7961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:37911
I20260809 09:31:29.271149 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:31:29.279351 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.272891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52082 (local address 127.9.102.62:39447)
0809 09:31:29.273011 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:29.273014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:29.273180 (+ 166us) server_negotiation.cc:407] Connection header received
0809 09:31:29.273254 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:29.273256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:29.273287 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:29.273343 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:29.273927 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.274349 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.275044 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.275218 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.275282 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.276616 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:29.276618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:29.276620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:29.276639 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:29.277941 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:29.278329 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:29.278331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:29.278332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:29.278398 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:29.278655 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:29.278657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:29.278658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:29.278813 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:29.278913 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:29.279127 (+ 214us) server_negotiation.cc:299] Negotiation successful
0809 09:31:29.279192 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 09:31:29.281862 7961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:31:29.284557 7961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:31:29.297152 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.291933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.9.102.3:38141)
0809 09:31:29.292370 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:29.292384 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:29.292415 (+ 31us) server_negotiation.cc:407] Connection header received
0809 09:31:29.292477 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:29.292482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:29.292682 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:29.292832 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:29.293152 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.294135 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.295730 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.296507 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.296646 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.296773 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:29.296950 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 09:31:29.297041 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":148,"threads_started":1}
I20260809 09:31:29.297150 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.291825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:38141 (local address 127.0.0.1:43982)
0809 09:31:29.292172 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:29.292216 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:29.292335 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:29.292843 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:29.292848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:29.292872 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:29.293011 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.293022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.294263 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.294266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.295606 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.295616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.296641 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.296646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.296752 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.296789 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 09:31:29.296916 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 09:31:29.297018 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":60,"thread_start_us":71,"threads_started":1}
I20260809 09:31:29.298055 8792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.292302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:33845 (local address 127.0.0.1:44352)
0809 09:31:29.293079 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:29.293091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:29.293298 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:29.293910 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:29.293912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:29.293925 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:29.293983 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.293986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.295348 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.295351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.296425 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.296435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.297711 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.297714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.297792 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.297812 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 09:31:29.297887 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 09:31:29.297939 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":38,"thread_start_us":93,"threads_started":1}
I20260809 09:31:29.298426 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.292453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.9.102.1:33845)
0809 09:31:29.292782 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:29.292794 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:29.293233 (+ 439us) server_negotiation.cc:407] Connection header received
0809 09:31:29.293444 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:29.293450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:29.293651 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:29.293777 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:29.294107 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.295202 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.296581 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.297393 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.297494 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.297638 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:29.298241 (+ 603us) server_negotiation.cc:299] Negotiation successful
0809 09:31:29.298325 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":103,"threads_started":1}
I20260809 09:31:29.299918 8648 tablet_service.cc:1467] Processing CreateTablet for tablet 33641a7cc7544753a737c2a36586314f (DEFAULT_TABLE table=test-table [id=0f632d1e0cbe44dfb055d1094441fcda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:29.300848 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.293098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:34281 (local address 127.0.0.1:56828)
0809 09:31:29.293581 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:29.293593 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:29.293671 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:29.294391 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:29.294393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:29.294408 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:29.294524 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.294528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.295489 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.295492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.296795 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.296808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.300468 (+ 3660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.300473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.300569 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.300590 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 09:31:29.300672 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 09:31:29.300737 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":40,"thread_start_us":88,"threads_started":1}
I20260809 09:31:29.300925 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.293481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.9.102.2:34281)
0809 09:31:29.293839 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:29.293848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:29.293868 (+ 20us) server_negotiation.cc:407] Connection header received
0809 09:31:29.293929 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:29.293933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:29.294099 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:29.294242 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:29.294643 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.295348 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.297401 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.297972 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.298094 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.298220 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:29.300704 (+ 2484us) server_negotiation.cc:299] Negotiation successful
0809 09:31:29.300793 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260809 09:31:29.300583 8187 tablet_service.cc:1467] Processing CreateTablet for tablet 33641a7cc7544753a737c2a36586314f (DEFAULT_TABLE table=test-table [id=0f632d1e0cbe44dfb055d1094441fcda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:29.300985 8648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33641a7cc7544753a737c2a36586314f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:29.301406 8187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33641a7cc7544753a737c2a36586314f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:29.303627 8408 tablet_service.cc:1467] Processing CreateTablet for tablet 33641a7cc7544753a737c2a36586314f (DEFAULT_TABLE table=test-table [id=0f632d1e0cbe44dfb055d1094441fcda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:29.304636 8408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33641a7cc7544753a737c2a36586314f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:29.306630 8799 tablet_bootstrap.cc:492] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6: Bootstrap starting.
I20260809 09:31:29.307821 8800 tablet_bootstrap.cc:492] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda: Bootstrap starting.
I20260809 09:31:29.309372 8799 tablet_bootstrap.cc:654] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:29.310199 8800 tablet_bootstrap.cc:654] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:29.310216 8799 log.cc:824] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:29.310449 8801 tablet_bootstrap.cc:492] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00: Bootstrap starting.
I20260809 09:31:29.310973 8800 log.cc:824] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:29.312340 8799 tablet_bootstrap.cc:492] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6: No bootstrap required, opened a new log
I20260809 09:31:29.312515 8799 ts_tablet_manager.cc:1397] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 09:31:29.312660 8801 tablet_bootstrap.cc:654] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:29.312879 8800 tablet_bootstrap.cc:492] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda: No bootstrap required, opened a new log
I20260809 09:31:29.313059 8800 ts_tablet_manager.cc:1397] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:31:29.313366 8801 log.cc:824] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:29.315320 8801 tablet_bootstrap.cc:492] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00: No bootstrap required, opened a new log
I20260809 09:31:29.315498 8801 ts_tablet_manager.cc:1397] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:31:29.316126 8799 raft_consensus.cc:348] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1952653c894c44b4af6fb52ef96c3ba6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38141 } } peers { permanent_uuid: "b6c673fea27940e08cd7e0c8477c6b00" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34281 } } peers { permanent_uuid: "ce6499b4680a41f89bb50597aa705bda" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 33845 } }
I20260809 09:31:29.316390 8799 raft_consensus.cc:374] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:29.316443 8799 raft_consensus.cc:729] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1952653c894c44b4af6fb52ef96c3ba6, State: Initialized, Role: FOLLOWER
I20260809 09:31:29.316502 8800 raft_consensus.cc:348] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1952653c894c44b4af6fb52ef96c3ba6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38141 } } peers { permanent_uuid: "b6c673fea27940e08cd7e0c8477c6b00" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34281 } } peers { permanent_uuid: "ce6499b4680a41f89bb50597aa705bda" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 33845 } }
I20260809 09:31:29.316730 8800 raft_consensus.cc:374] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:29.316782 8800 raft_consensus.cc:729] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6499b4680a41f89bb50597aa705bda, State: Initialized, Role: FOLLOWER
I20260809 09:31:29.316865 8799 consensus_queue.cc:260] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1952653c894c44b4af6fb52ef96c3ba6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38141 } } peers { permanent_uuid: "b6c673fea27940e08cd7e0c8477c6b00" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34281 } } peers { permanent_uuid: "ce6499b4680a41f89bb50597aa705bda" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 33845 } }
I20260809 09:31:29.317157 8800 consensus_queue.cc:260] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1952653c894c44b4af6fb52ef96c3ba6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38141 } } peers { permanent_uuid: "b6c673fea27940e08cd7e0c8477c6b00" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34281 } } peers { permanent_uuid: "ce6499b4680a41f89bb50597aa705bda" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 33845 } }
I20260809 09:31:29.317489 8771 heartbeater.cc:503] Master 127.9.102.62:39447 was elected leader, sending a full tablet report...
I20260809 09:31:29.317762 8799 ts_tablet_manager.cc:1428] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:31:29.318316 8306 heartbeater.cc:503] Master 127.9.102.62:39447 was elected leader, sending a full tablet report...
I20260809 09:31:29.318723 8801 raft_consensus.cc:348] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1952653c894c44b4af6fb52ef96c3ba6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38141 } } peers { permanent_uuid: "b6c673fea27940e08cd7e0c8477c6b00" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34281 } } peers { permanent_uuid: "ce6499b4680a41f89bb50597aa705bda" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 33845 } }
I20260809 09:31:29.318929 8801 raft_consensus.cc:374] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:29.318982 8801 raft_consensus.cc:729] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c673fea27940e08cd7e0c8477c6b00, State: Initialized, Role: FOLLOWER
I20260809 09:31:29.319355 8801 consensus_queue.cc:260] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1952653c894c44b4af6fb52ef96c3ba6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38141 } } peers { permanent_uuid: "b6c673fea27940e08cd7e0c8477c6b00" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34281 } } peers { permanent_uuid: "ce6499b4680a41f89bb50597aa705bda" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 33845 } }
I20260809 09:31:29.319648 8800 ts_tablet_manager.cc:1428] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260809 09:31:29.319952 8531 heartbeater.cc:503] Master 127.9.102.62:39447 was elected leader, sending a full tablet report...
I20260809 09:31:29.320732 8801 ts_tablet_manager.cc:1428] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260809 09:31:29.361065 8307 tablet.cc:2369] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:31:29.495967 8773 tablet.cc:2369] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:31:29.535014 8810 raft_consensus.cc:482] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:31:29.535226 8810 raft_consensus.cc:504] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1952653c894c44b4af6fb52ef96c3ba6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38141 } } peers { permanent_uuid: "b6c673fea27940e08cd7e0c8477c6b00" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34281 } } peers { permanent_uuid: "ce6499b4680a41f89bb50597aa705bda" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 33845 } }
I20260809 09:31:29.536405 8810 leader_election.cc:288] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1952653c894c44b4af6fb52ef96c3ba6 (127.9.102.3:38141), ce6499b4680a41f89bb50597aa705bda (127.9.102.1:33845)
I20260809 09:31:29.540019 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.536713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:51159 (local address 127.9.102.1:33845)
0809 09:31:29.536833 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:29.536838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:29.537147 (+ 309us) server_negotiation.cc:407] Connection header received
0809 09:31:29.537265 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:29.537269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:29.537317 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:29.537394 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:29.537667 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.538125 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.539110 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.539450 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.539574 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.539639 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:29.539852 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 09:31:29.539925 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 09:31:29.540009 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.536640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:38141 (local address 127.9.102.2:38881)
0809 09:31:29.536832 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:29.536847 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:29.536945 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:29.537248 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:29.537251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:29.537267 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:29.537395 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.537401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.538133 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.538136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.538910 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.538919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.539558 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.539573 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.539692 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.539720 (+ 28us) client_negotiation.cc:769] Sending connection context
0809 09:31:29.539777 (+ 57us) client_negotiation.cc:240] Negotiation successful
0809 09:31:29.539854 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260809 09:31:29.540007 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.536719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:38881 (local address 127.9.102.3:38141)
0809 09:31:29.536833 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:29.536838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:29.536903 (+ 65us) server_negotiation.cc:407] Connection header received
0809 09:31:29.537068 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:29.537071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:29.537129 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:29.537205 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:29.537501 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.538019 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.539046 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.539432 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.539507 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.539614 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:29.539823 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 09:31:29.539888 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 09:31:29.540024 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.536640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:33845 (local address 127.9.102.2:51159)
0809 09:31:29.537042 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:29.537056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:29.537164 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:29.537412 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:29.537414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:29.537428 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:29.537560 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.537566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.538238 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.538240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.538992 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:29.539005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.539607 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.539610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:29.539720 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.539736 (+ 16us) client_negotiation.cc:769] Sending connection context
0809 09:31:29.539825 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 09:31:29.539889 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":51,"thread_start_us":77,"threads_started":1}
I20260809 09:31:29.540655 8222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33641a7cc7544753a737c2a36586314f" candidate_uuid: "b6c673fea27940e08cd7e0c8477c6b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6499b4680a41f89bb50597aa705bda" is_pre_election: true
I20260809 09:31:29.540693 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33641a7cc7544753a737c2a36586314f" candidate_uuid: "b6c673fea27940e08cd7e0c8477c6b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1952653c894c44b4af6fb52ef96c3ba6" is_pre_election: true
I20260809 09:31:29.540946 8222 raft_consensus.cc:2454] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c673fea27940e08cd7e0c8477c6b00 in term 0.
I20260809 09:31:29.540969 8690 raft_consensus.cc:2454] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c673fea27940e08cd7e0c8477c6b00 in term 0.
I20260809 09:31:29.541381 8345 leader_election.cc:302] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c673fea27940e08cd7e0c8477c6b00, ce6499b4680a41f89bb50597aa705bda; no voters:
I20260809 09:31:29.541640 8810 raft_consensus.cc:2781] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:31:29.541718 8810 raft_consensus.cc:482] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:31:29.541766 8810 raft_consensus.cc:3037] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:29.542721 8810 raft_consensus.cc:504] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1952653c894c44b4af6fb52ef96c3ba6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38141 } } peers { permanent_uuid: "b6c673fea27940e08cd7e0c8477c6b00" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34281 } } peers { permanent_uuid: "ce6499b4680a41f89bb50597aa705bda" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 33845 } }
I20260809 09:31:29.543063 8810 leader_election.cc:288] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [CANDIDATE]: Term 1 election: Requested vote from peers 1952653c894c44b4af6fb52ef96c3ba6 (127.9.102.3:38141), ce6499b4680a41f89bb50597aa705bda (127.9.102.1:33845)
I20260809 09:31:29.543288 8690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33641a7cc7544753a737c2a36586314f" candidate_uuid: "b6c673fea27940e08cd7e0c8477c6b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1952653c894c44b4af6fb52ef96c3ba6"
I20260809 09:31:29.543408 8222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33641a7cc7544753a737c2a36586314f" candidate_uuid: "b6c673fea27940e08cd7e0c8477c6b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6499b4680a41f89bb50597aa705bda"
I20260809 09:31:29.543417 8690 raft_consensus.cc:3037] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:29.543511 8222 raft_consensus.cc:3037] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:29.544658 8222 raft_consensus.cc:2454] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c673fea27940e08cd7e0c8477c6b00 in term 1.
I20260809 09:31:29.544658 8690 raft_consensus.cc:2454] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c673fea27940e08cd7e0c8477c6b00 in term 1.
I20260809 09:31:29.544938 8345 leader_election.cc:302] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6c673fea27940e08cd7e0c8477c6b00, ce6499b4680a41f89bb50597aa705bda; no voters:
I20260809 09:31:29.545087 8810 raft_consensus.cc:2781] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:29.545307 8810 raft_consensus.cc:686] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 1 LEADER]: Becoming Leader. State: Replica: b6c673fea27940e08cd7e0c8477c6b00, State: Running, Role: LEADER
I20260809 09:31:29.545604 8810 consensus_queue.cc:237] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1952653c894c44b4af6fb52ef96c3ba6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38141 } } peers { permanent_uuid: "b6c673fea27940e08cd7e0c8477c6b00" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34281 } } peers { permanent_uuid: "ce6499b4680a41f89bb50597aa705bda" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 33845 } }
I20260809 09:31:29.548650 7960 catalog_manager.cc:5495] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6c673fea27940e08cd7e0c8477c6b00 (127.9.102.2). New cstate: current_term: 1 leader_uuid: "b6c673fea27940e08cd7e0c8477c6b00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1952653c894c44b4af6fb52ef96c3ba6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6c673fea27940e08cd7e0c8477c6b00" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 34281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce6499b4680a41f89bb50597aa705bda" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 33845 } health_report { overall_health: UNKNOWN } } }
W20260809 09:31:29.554919 8532 tablet.cc:2369] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:31:29.587404 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.584148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.9.102.2:34281)
0809 09:31:29.584274 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:29.584278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:29.584291 (+ 13us) server_negotiation.cc:407] Connection header received
0809 09:31:29.584436 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:29.584438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:29.584485 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:29.584584 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:31:29.584833 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.585270 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.586001 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.586183 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.586301 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.586623 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:31:29.586987 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:31:29.587135 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:29.587227 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 09:31:29.587264 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:31:29.592536 8690 raft_consensus.cc:1261] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6 [term 1 FOLLOWER]: Refusing update from remote peer b6c673fea27940e08cd7e0c8477c6b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:31:29.592536 8222 raft_consensus.cc:1261] T 33641a7cc7544753a737c2a36586314f P ce6499b4680a41f89bb50597aa705bda [term 1 FOLLOWER]: Refusing update from remote peer b6c673fea27940e08cd7e0c8477c6b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:31:29.593176 8810 consensus_queue.cc:1035] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce6499b4680a41f89bb50597aa705bda" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 33845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:31:29.593321 8812 consensus_queue.cc:1035] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1952653c894c44b4af6fb52ef96c3ba6" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:31:29.602051 8823 mvcc.cc:205] Tried to move back new op lower bound from 7316553275764871168 to 7316553275581480960. Current Snapshot: MvccSnapshot[applied={T|T < 7316553275764871168}]
I20260809 09:31:29.604058 8824 mvcc.cc:205] Tried to move back new op lower bound from 7316553275764871168 to 7316553275581480960. Current Snapshot: MvccSnapshot[applied={T|T < 7316553275764871168}]
I20260809 09:31:29.611624 7960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:31:29.611773 7960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:31:29.613986 7960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70619fe3c9ed471bb7d3d7fde46c043a: Sending DeleteTablet for 3 replicas of tablet 33641a7cc7544753a737c2a36586314f
I20260809 09:31:29.614727 8408 tablet_service.cc:1514] Processing DeleteTablet for tablet 33641a7cc7544753a737c2a36586314f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:29 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:56828
I20260809 09:31:29.614750 8187 tablet_service.cc:1514] Processing DeleteTablet for tablet 33641a7cc7544753a737c2a36586314f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:29 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:44352
I20260809 09:31:29.615216 8841 tablet_replica.cc:330] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00: stopping tablet replica
I20260809 09:31:29.615252 8648 tablet_service.cc:1514] Processing DeleteTablet for tablet 33641a7cc7544753a737c2a36586314f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:29 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:43982
I20260809 09:31:29.615511 8841 raft_consensus.cc:2229] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:31:29.615866 8845 tablet_replica.cc:330] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6: stopping tablet replica
I20260809 09:31:29.615867 8841 raft_consensus.cc:2258] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:29.616173 8845 raft_consensus.cc:2229] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:31:29.616408 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 8075
I20260809 09:31:29.616494 8845 raft_consensus.cc:2258] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:29.617801 8841 ts_tablet_manager.cc:1905] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:31:29.618312 8845 ts_tablet_manager.cc:1905] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:31:29.621053 8845 ts_tablet_manager.cc:1918] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:31:29.621181 8845 log.cc:1196] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/33641a7cc7544753a737c2a36586314f
I20260809 09:31:29.621543 8841 ts_tablet_manager.cc:1918] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:31:29.621567 8845 ts_tablet_manager.cc:1939] T 33641a7cc7544753a737c2a36586314f P 1952653c894c44b4af6fb52ef96c3ba6: Deleting consensus metadata
I20260809 09:31:29.621634 8841 log.cc:1196] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/33641a7cc7544753a737c2a36586314f
I20260809 09:31:29.621989 8841 ts_tablet_manager.cc:1939] T 33641a7cc7544753a737c2a36586314f P b6c673fea27940e08cd7e0c8477c6b00: Deleting consensus metadata
I20260809 09:31:29.622655 7940 catalog_manager.cc:4841] TS 1952653c894c44b4af6fb52ef96c3ba6 (127.9.102.3:38141): tablet 33641a7cc7544753a737c2a36586314f (table test-table [id=0f632d1e0cbe44dfb055d1094441fcda]) successfully deleted
I20260809 09:31:29.624967 7940 catalog_manager.cc:4841] TS b6c673fea27940e08cd7e0c8477c6b00 (127.9.102.2:34281): tablet 33641a7cc7544753a737c2a36586314f (table test-table [id=0f632d1e0cbe44dfb055d1094441fcda]) successfully deleted
W20260809 09:31:29.631152 7937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:33845 (error 108)
I20260809 09:31:29.631189 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 8313
I20260809 09:31:29.636849 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.636555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:33845 (local address 127.0.0.1:35352)
0809 09:31:29.636708 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:29.636782 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 09:31:29.637046 7937 catalog_manager.cc:4584] TS ce6499b4680a41f89bb50597aa705bda (127.9.102.1:33845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33641a7cc7544753a737c2a36586314f: Network error: Client connection negotiation failed: client connection to 127.9.102.1:33845: connect: Connection refused (error 111)
I20260809 09:31:29.640679 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 8537
I20260809 09:31:29.649537 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 7904
2026-08-09T09:31:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2491 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T09:31:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:29Z Disabled control of system clock
I20260809 09:31:29.685557 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:38397
--webserver_interface=127.9.102.62
--webserver_port=0
--builtin_ntp_servers=127.9.102.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:38397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 09:31:29.782589 8854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:29.782778 8854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:29.782807 8854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:29.786222 8854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:29.786264 8854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:29.786278 8854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:29.786291 8854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:29.786303 8854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:29.790210 8854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:38397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:38397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:29.790951 8854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:29.791993 8854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:29.799314 8864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:29.799332 8861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:29.799332 8870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:29.799373 8854 server_base.cc:1034] running on GCE node
I20260809 09:31:29.800068 8854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:29.801039 8854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:29.802271 8854 hybrid_clock.cc:648] HybridClock initialized: now 1786267889802255 us; error 32 us; skew 500 ppm
I20260809 09:31:29.803861 8854 webserver.cc:460] Webserver started at http://127.9.102.62:37803/ using document root <none> and password file <none>
I20260809 09:31:29.804447 8854 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:29.804535 8854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:29.804733 8854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:29.806495 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "897ed7fa570f467590470f0b997d9ebb"
format_stamp: "Formatted at 2026-08-09 09:31:29 on dist-test-slave-7vjv"
server_key: "88c66ec5452458052ffb9a3369c65b8d"
server_key_iv: "76669ae50a722d470402aba9d640b22e"
server_key_version: "encryptionkey@0"
I20260809 09:31:29.807031 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "897ed7fa570f467590470f0b997d9ebb"
format_stamp: "Formatted at 2026-08-09 09:31:29 on dist-test-slave-7vjv"
server_key: "88c66ec5452458052ffb9a3369c65b8d"
server_key_iv: "76669ae50a722d470402aba9d640b22e"
server_key_version: "encryptionkey@0"
I20260809 09:31:29.810297 8854 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:31:29.812325 8885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:29.813365 8854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 09:31:29.813498 8854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "897ed7fa570f467590470f0b997d9ebb"
format_stamp: "Formatted at 2026-08-09 09:31:29 on dist-test-slave-7vjv"
server_key: "88c66ec5452458052ffb9a3369c65b8d"
server_key_iv: "76669ae50a722d470402aba9d640b22e"
server_key_version: "encryptionkey@0"
I20260809 09:31:29.813591 8854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:29.829332 8854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:29.829825 8854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:29.829988 8854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:29.836621 8854 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:38397
I20260809 09:31:29.836630 8989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:38397 every 8 connection(s)
I20260809 09:31:29.837396 8854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:29.840047 8992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:29.840904 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 8854
I20260809 09:31:29.841017 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:29.841261 9624 external_mini_cluster.cc:1428] Setting key a2ec44ef6f0e722f05d1b01943ec71a7
I20260809 09:31:29.847530 8992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb: Bootstrap starting.
I20260809 09:31:29.849602 8992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:29.850371 8992 log.cc:824] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:29.850600 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:29.842424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.9.102.62:38397)
0809 09:31:29.843077 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:29.843098 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:29.843139 (+ 41us) server_negotiation.cc:407] Connection header received
0809 09:31:29.843976 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:29.843999 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:29.844488 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:29.844844 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:29.845546 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.846912 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.847680 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:29.848181 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:29.848344 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:29.849129 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:29.849134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:29.849145 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:29.849467 (+ 322us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:29.849513 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:29.849533 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:29.849634 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:29.849743 (+ 109us) server_negotiation.cc:299] Negotiation successful
0809 09:31:29.849973 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":103,"threads_started":1}
I20260809 09:31:29.852010 8992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb: No bootstrap required, opened a new log
I20260809 09:31:29.854809 8992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897ed7fa570f467590470f0b997d9ebb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38397 } }
I20260809 09:31:29.854982 8992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:29.855041 8992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897ed7fa570f467590470f0b997d9ebb, State: Initialized, Role: FOLLOWER
I20260809 09:31:29.855407 8992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897ed7fa570f467590470f0b997d9ebb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38397 } }
I20260809 09:31:29.855512 8992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:29.855592 8992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:29.855705 8992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:29.856469 8992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897ed7fa570f467590470f0b997d9ebb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38397 } }
I20260809 09:31:29.856765 8992 leader_election.cc:302] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 897ed7fa570f467590470f0b997d9ebb; no voters:
I20260809 09:31:29.857033 8992 leader_election.cc:288] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:29.857144 9004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:29.857367 9004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [term 1 LEADER]: Becoming Leader. State: Replica: 897ed7fa570f467590470f0b997d9ebb, State: Running, Role: LEADER
I20260809 09:31:29.857609 9004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897ed7fa570f467590470f0b997d9ebb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38397 } }
I20260809 09:31:29.858121 8992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:29.859117 9005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "897ed7fa570f467590470f0b997d9ebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897ed7fa570f467590470f0b997d9ebb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38397 } } }
I20260809 09:31:29.859123 9006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 897ed7fa570f467590470f0b997d9ebb. Latest consensus state: current_term: 1 leader_uuid: "897ed7fa570f467590470f0b997d9ebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897ed7fa570f467590470f0b997d9ebb" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 38397 } } }
I20260809 09:31:29.859252 9005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:29.859251 9006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:29.859562 9020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:29.861719 9020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:29.866451 9020 catalog_manager.cc:1347] Generated new cluster ID: e45fa03b20c343e48a5dda862c30923b
I20260809 09:31:29.866505 9020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:29.888696 9020 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:29.889732 9020 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:29.897372 9020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb: Generated new TSK 0
I20260809 09:31:29.897928 9020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:29.908444 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38397
--builtin_ntp_servers=127.9.102.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 09:31:30.007787 9037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:30.007987 9037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:30.008018 9037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:30.011372 9037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:30.011413 9037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:30.011474 9037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:31:30.015440 9037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:30.016307 9037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:30.017513 9037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:30.025961 9055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:30.025988 9051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:30.026160 9037 server_base.cc:1034] running on GCE node
W20260809 09:31:30.026188 9048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:30.026552 9037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:30.027104 9037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:30.028288 9037 hybrid_clock.cc:648] HybridClock initialized: now 1786267890028268 us; error 38 us; skew 500 ppm
I20260809 09:31:30.029922 9037 webserver.cc:460] Webserver started at http://127.9.102.1:40455/ using document root <none> and password file <none>
I20260809 09:31:30.030476 9037 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:30.030570 9037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:30.030783 9037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:30.032620 9037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "e2c6f567d7d948f8b1ab1739ff94c779"
format_stamp: "Formatted at 2026-08-09 09:31:30 on dist-test-slave-7vjv"
server_key: "a0787ed914f176252d9de7b516383cc5"
server_key_iv: "1b1b67d4efe7a740a5a2e0e086ee50ed"
server_key_version: "encryptionkey@0"
I20260809 09:31:30.033171 9037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "e2c6f567d7d948f8b1ab1739ff94c779"
format_stamp: "Formatted at 2026-08-09 09:31:30 on dist-test-slave-7vjv"
server_key: "a0787ed914f176252d9de7b516383cc5"
server_key_iv: "1b1b67d4efe7a740a5a2e0e086ee50ed"
server_key_version: "encryptionkey@0"
I20260809 09:31:30.036556 9037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:31:30.038695 9065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:30.039723 9037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:30.039865 9037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "e2c6f567d7d948f8b1ab1739ff94c779"
format_stamp: "Formatted at 2026-08-09 09:31:30 on dist-test-slave-7vjv"
server_key: "a0787ed914f176252d9de7b516383cc5"
server_key_iv: "1b1b67d4efe7a740a5a2e0e086ee50ed"
server_key_version: "encryptionkey@0"
I20260809 09:31:30.039989 9037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:30.056264 9037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:30.056787 9037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:30.056962 9037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:30.057448 9037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:30.058298 9037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:30.058362 9037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:30.058461 9037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:30.058509 9037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:30.066548 9037 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:42709
I20260809 09:31:30.066610 9261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:42709 every 8 connection(s)
I20260809 09:31:30.067337 9037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:31:30.074505 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 9037
I20260809 09:31:30.074640 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:31:30.074976 9624 external_mini_cluster.cc:1428] Setting key 8a5254f33edb5c0f07b7cd9f3c1216ef
I20260809 09:31:30.075204 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.068927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:48015 (local address 127.9.102.62:38397)
0809 09:31:30.069064 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:30.069068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:30.069882 (+ 814us) server_negotiation.cc:407] Connection header received
0809 09:31:30.071037 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:30.071040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:30.071083 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:30.071149 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:30.071867 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.072429 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.073421 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.073606 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.073674 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.074696 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:30.074700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:30.074703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:30.074716 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:30.074739 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:30.074749 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:30.074833 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:30.074972 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 09:31:30.075044 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 09:31:30.075973 9267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.069230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38397 (local address 127.9.102.1:48015)
0809 09:31:30.069777 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:30.069810 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:30.070866 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:30.071293 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:30.071298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:30.071370 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:30.071757 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.071767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.072533 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.072536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.073325 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.073333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.073729 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.073732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.073848 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.074422 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:31:30.074423 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:30.074458 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:31:30.074513 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:30.074841 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:30.074849 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:30.074853 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 09:31:30.074941 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 09:31:30.075202 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":106,"threads_started":1}
I20260809 09:31:30.076949 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38397
--builtin_ntp_servers=127.9.102.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 09:31:30.077047 9265 heartbeater.cc:347] Connected to a master server at 127.9.102.62:38397
I20260809 09:31:30.077217 9265 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:30.077649 9265 heartbeater.cc:511] Master 127.9.102.62:38397 requested a full tablet report, sending...
I20260809 09:31:30.079151 8922 ts_manager.cc:194] Registered new tserver with Master: e2c6f567d7d948f8b1ab1739ff94c779 (127.9.102.1:42709)
I20260809 09:31:30.080690 8922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.102.1:48015
W20260809 09:31:30.176527 9269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:30.176721 9269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:30.176753 9269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:30.180271 9269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:30.180307 9269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:30.180351 9269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:31:30.184058 9269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:30.184756 9269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:30.185839 9269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:30.193708 9275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:30.193750 9274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:30.193994 9269 server_base.cc:1034] running on GCE node
W20260809 09:31:30.193723 9281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:30.194356 9269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:30.194936 9269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:30.196157 9269 hybrid_clock.cc:648] HybridClock initialized: now 1786267890196132 us; error 33 us; skew 500 ppm
I20260809 09:31:30.197824 9269 webserver.cc:460] Webserver started at http://127.9.102.2:34209/ using document root <none> and password file <none>
I20260809 09:31:30.198339 9269 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:30.198429 9269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:30.198638 9269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:30.200392 9269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "83289c6d98ce484786dfc77961e5884b"
format_stamp: "Formatted at 2026-08-09 09:31:30 on dist-test-slave-7vjv"
server_key: "65c752beebfc263a8b7d212cc4480ed3"
server_key_iv: "e950b898773c4ffcd4825643a2c403cb"
server_key_version: "encryptionkey@0"
I20260809 09:31:30.200918 9269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "83289c6d98ce484786dfc77961e5884b"
format_stamp: "Formatted at 2026-08-09 09:31:30 on dist-test-slave-7vjv"
server_key: "65c752beebfc263a8b7d212cc4480ed3"
server_key_iv: "e950b898773c4ffcd4825643a2c403cb"
server_key_version: "encryptionkey@0"
I20260809 09:31:30.204006 9269 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 09:31:30.206137 9292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:30.207180 9269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 09:31:30.207295 9269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "83289c6d98ce484786dfc77961e5884b"
format_stamp: "Formatted at 2026-08-09 09:31:30 on dist-test-slave-7vjv"
server_key: "65c752beebfc263a8b7d212cc4480ed3"
server_key_iv: "e950b898773c4ffcd4825643a2c403cb"
server_key_version: "encryptionkey@0"
I20260809 09:31:30.207389 9269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:30.235455 9269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:30.236003 9269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:30.236207 9269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:30.236720 9269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:30.237516 9269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:30.237571 9269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:30.237633 9269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:30.237672 9269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:30.245548 9269 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:45185
I20260809 09:31:30.245570 9498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:45185 every 8 connection(s)
I20260809 09:31:30.246272 9269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:31:30.253614 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.247808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:50709 (local address 127.9.102.62:38397)
0809 09:31:30.247922 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:30.247926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:30.248767 (+ 841us) server_negotiation.cc:407] Connection header received
0809 09:31:30.249897 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:30.249899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:30.249942 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:30.250009 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:30.250755 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.251190 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.252134 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.252320 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.252390 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.253249 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:30.253252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:30.253254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:30.253264 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:30.253281 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:30.253288 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:30.253342 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:30.253468 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 09:31:30.253513 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:31:30.254292 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.248080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38397 (local address 127.9.102.2:50709)
0809 09:31:30.248635 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:30.248668 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:30.249733 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:30.250147 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:30.250152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:30.250224 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:30.250645 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.250655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.251285 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.251287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.251999 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.252007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.252432 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.252434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.252558 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.253034 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:31:30.253035 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:30.253068 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:31:30.253119 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:30.253357 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:30.253363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:30.253367 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 09:31:30.253447 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 09:31:30.253624 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
I20260809 09:31:30.254913 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 9269
I20260809 09:31:30.255059 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:31:30.255336 9624 external_mini_cluster.cc:1428] Setting key 4fed7894c1d60c10a1570b06ee6224f9
I20260809 09:31:30.255409 9499 heartbeater.cc:347] Connected to a master server at 127.9.102.62:38397
I20260809 09:31:30.255617 9499 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:30.256134 9499 heartbeater.cc:511] Master 127.9.102.62:38397 requested a full tablet report, sending...
I20260809 09:31:30.257100 8922 ts_manager.cc:194] Registered new tserver with Master: 83289c6d98ce484786dfc77961e5884b (127.9.102.2:45185)
I20260809 09:31:30.257329 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38397
--builtin_ntp_servers=127.9.102.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 09:31:30.257747 8922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.102.2:50709
W20260809 09:31:30.362529 9503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:30.362746 9503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:30.362804 9503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:30.366284 9503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:30.366348 9503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:30.366420 9503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:31:30.370587 9503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:30.371472 9503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:30.372601 9503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:30.380834 9515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:30.380841 9513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:30.380888 9503 server_base.cc:1034] running on GCE node
W20260809 09:31:30.380841 9519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:30.381430 9503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:30.381956 9503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:30.383132 9503 hybrid_clock.cc:648] HybridClock initialized: now 1786267890383122 us; error 34 us; skew 500 ppm
I20260809 09:31:30.384757 9503 webserver.cc:460] Webserver started at http://127.9.102.3:36239/ using document root <none> and password file <none>
I20260809 09:31:30.385250 9503 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:30.385336 9503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:30.385547 9503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:30.387266 9503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "2387d5bdca57450fb3df77183cc13423"
format_stamp: "Formatted at 2026-08-09 09:31:30 on dist-test-slave-7vjv"
server_key: "1206211849d33de54d58ac269d788616"
server_key_iv: "a1a074b67676fee5dcf237408388af97"
server_key_version: "encryptionkey@0"
I20260809 09:31:30.387773 9503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "2387d5bdca57450fb3df77183cc13423"
format_stamp: "Formatted at 2026-08-09 09:31:30 on dist-test-slave-7vjv"
server_key: "1206211849d33de54d58ac269d788616"
server_key_iv: "a1a074b67676fee5dcf237408388af97"
server_key_version: "encryptionkey@0"
I20260809 09:31:30.391124 9503 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:31:30.393352 9528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:30.394397 9503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:31:30.394526 9503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "2387d5bdca57450fb3df77183cc13423"
format_stamp: "Formatted at 2026-08-09 09:31:30 on dist-test-slave-7vjv"
server_key: "1206211849d33de54d58ac269d788616"
server_key_iv: "a1a074b67676fee5dcf237408388af97"
server_key_version: "encryptionkey@0"
I20260809 09:31:30.394629 9503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:30.406288 9503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:30.406833 9503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:30.406998 9503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:30.407464 9503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:30.408289 9503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:30.408353 9503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:30.408450 9503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:30.408496 9503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:30.416337 9503 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:39273
I20260809 09:31:30.416368 9738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:39273 every 8 connection(s)
I20260809 09:31:30.417124 9503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:31:30.424674 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.418642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:43967 (local address 127.9.102.62:38397)
0809 09:31:30.418787 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:30.418791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:30.419691 (+ 900us) server_negotiation.cc:407] Connection header received
0809 09:31:30.420936 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:30.420938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:30.420980 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:30.421046 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:30.421740 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.422170 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.423094 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.423277 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.423363 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.424253 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:30.424256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:30.424258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:30.424268 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:30.424284 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:30.424291 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:30.424346 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:30.424469 (+ 123us) server_negotiation.cc:299] Negotiation successful
0809 09:31:30.424511 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 09:31:30.425549 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.418974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:38397 (local address 127.9.102.3:43967)
0809 09:31:30.419546 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:30.419580 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:30.420755 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:30.421182 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:30.421187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:30.421254 (+ 67us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:30.421626 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.421636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.422268 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.422270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.422991 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.422999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.423406 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.423408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.423528 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.423978 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:31:30.423980 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:30.424011 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:31:30.424070 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:30.424361 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:30.424367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:30.424371 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 09:31:30.424452 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 09:31:30.424652 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":95,"threads_started":1}
I20260809 09:31:30.426128 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 9503
I20260809 09:31:30.426246 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:31:30.426532 9624 external_mini_cluster.cc:1428] Setting key 382c0b3263f917cf6772860cb752ac3c
I20260809 09:31:30.426680 9739 heartbeater.cc:347] Connected to a master server at 127.9.102.62:38397
I20260809 09:31:30.426890 9739 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:30.427445 9739 heartbeater.cc:511] Master 127.9.102.62:38397 requested a full tablet report, sending...
I20260809 09:31:30.428428 8921 ts_manager.cc:194] Registered new tserver with Master: 2387d5bdca57450fb3df77183cc13423 (127.9.102.3:39273)
I20260809 09:31:30.428913 8921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.102.3:43967
I20260809 09:31:30.440168 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:31:30.445225 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.441913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.9.102.62:38397)
0809 09:31:30.442010 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:30.442013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:30.442158 (+ 145us) server_negotiation.cc:407] Connection header received
0809 09:31:30.442229 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:30.442231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:30.442259 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:30.442317 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:30.442540 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.443007 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.443801 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.444092 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.444196 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.444884 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:30.444886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:30.444888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:30.444898 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:30.444912 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:30.444919 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:30.445007 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:30.445117 (+ 110us) server_negotiation.cc:299] Negotiation successful
0809 09:31:30.445141 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260809 09:31:30.447644 8921 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:31:30.450350 8921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:31:30.462889 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.457401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:42709 (local address 127.0.0.1:37420)
0809 09:31:30.458002 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:30.458030 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:30.458161 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:30.458661 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:30.458666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:30.458686 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:30.458805 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.458811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.460270 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.460273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.461170 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.461180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.462447 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.462450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.462561 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.462597 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 09:31:30.462691 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 09:31:30.462780 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":72,"threads_started":1}
I20260809 09:31:30.463151 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.458432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.9.102.3:39273)
0809 09:31:30.458709 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:30.458719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:30.458738 (+ 19us) server_negotiation.cc:407] Connection header received
0809 09:31:30.458795 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:30.458800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:30.458953 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:30.459079 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:30.459327 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.460336 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.461707 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.462362 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.462450 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.462609 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:30.462907 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 09:31:30.463002 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260809 09:31:30.463475 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.457515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.9.102.1:42709)
0809 09:31:30.457857 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:30.457869 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:30.458128 (+ 259us) server_negotiation.cc:407] Connection header received
0809 09:31:30.458312 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:30.458318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:30.458503 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:30.458938 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:30.459089 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.460164 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.461418 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.462338 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.462451 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.462609 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:30.462716 (+ 107us) server_negotiation.cc:299] Negotiation successful
0809 09:31:30.462805 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":163,"threads_started":1}
I20260809 09:31:30.463581 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.457508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:39273 (local address 127.0.0.1:48180)
0809 09:31:30.458388 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:30.458398 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:30.458463 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:30.459077 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:30.459080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:30.459095 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:30.459178 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.459183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.460457 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.460460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.461572 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.461580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.462673 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.462677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.462758 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.462779 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 09:31:30.463335 (+ 556us) client_negotiation.cc:240] Negotiation successful
0809 09:31:30.463403 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"thread_start_us":86,"threads_started":1}
I20260809 09:31:30.463598 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.458062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:45185 (local address 127.0.0.1:37762)
0809 09:31:30.458740 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:30.458752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:30.458825 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:30.459826 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:30.459829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:30.459849 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:30.459952 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.459957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.461377 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.461380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.462259 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.462268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.463206 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.463209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.463282 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.463312 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 09:31:30.463367 (+ 55us) client_negotiation.cc:240] Negotiation successful
0809 09:31:30.463448 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":98,"threads_started":1}
I20260809 09:31:30.463653 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.458222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.9.102.2:45185)
0809 09:31:30.459347 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:30.459376 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:30.459399 (+ 23us) server_negotiation.cc:407] Connection header received
0809 09:31:30.459466 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:30.459471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:30.459689 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:30.459820 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:30.460113 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.461244 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.462378 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.463091 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.463203 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.463374 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:30.463414 (+ 40us) server_negotiation.cc:299] Negotiation successful
0809 09:31:30.463514 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":128,"threads_started":1}
I20260809 09:31:30.466055 9614 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9a6ccd68ec470e8886555393f35270 (DEFAULT_TABLE table=test-table [id=310b16afdf37442ab2bfc4810d334709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:30.466192 9154 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9a6ccd68ec470e8886555393f35270 (DEFAULT_TABLE table=test-table [id=310b16afdf37442ab2bfc4810d334709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:30.466893 9614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9a6ccd68ec470e8886555393f35270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:30.466952 9154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9a6ccd68ec470e8886555393f35270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:30.472354 9765 tablet_bootstrap.cc:492] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779: Bootstrap starting.
I20260809 09:31:30.472395 9768 tablet_bootstrap.cc:492] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423: Bootstrap starting.
I20260809 09:31:30.473995 9381 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9a6ccd68ec470e8886555393f35270 (DEFAULT_TABLE table=test-table [id=310b16afdf37442ab2bfc4810d334709]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:30.474581 9768 tablet_bootstrap.cc:654] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:30.474592 9765 tablet_bootstrap.cc:654] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:30.474924 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9a6ccd68ec470e8886555393f35270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:30.475297 9765 log.cc:824] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:30.475338 9768 log.cc:824] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:30.477154 9768 tablet_bootstrap.cc:492] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423: No bootstrap required, opened a new log
I20260809 09:31:30.477154 9765 tablet_bootstrap.cc:492] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779: No bootstrap required, opened a new log
I20260809 09:31:30.477329 9768 ts_tablet_manager.cc:1397] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 09:31:30.477329 9765 ts_tablet_manager.cc:1397] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 09:31:30.480621 9773 tablet_bootstrap.cc:492] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b: Bootstrap starting.
I20260809 09:31:30.481312 9765 raft_consensus.cc:348] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 42709 } } peers { permanent_uuid: "2387d5bdca57450fb3df77183cc13423" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 39273 } } peers { permanent_uuid: "83289c6d98ce484786dfc77961e5884b" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45185 } }
I20260809 09:31:30.481315 9768 raft_consensus.cc:348] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 42709 } } peers { permanent_uuid: "2387d5bdca57450fb3df77183cc13423" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 39273 } } peers { permanent_uuid: "83289c6d98ce484786dfc77961e5884b" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45185 } }
I20260809 09:31:30.481547 9765 raft_consensus.cc:374] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:30.481559 9768 raft_consensus.cc:374] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:30.481607 9765 raft_consensus.cc:729] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c6f567d7d948f8b1ab1739ff94c779, State: Initialized, Role: FOLLOWER
I20260809 09:31:30.481607 9768 raft_consensus.cc:729] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2387d5bdca57450fb3df77183cc13423, State: Initialized, Role: FOLLOWER
I20260809 09:31:30.482052 9765 consensus_queue.cc:260] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 42709 } } peers { permanent_uuid: "2387d5bdca57450fb3df77183cc13423" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 39273 } } peers { permanent_uuid: "83289c6d98ce484786dfc77961e5884b" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45185 } }
I20260809 09:31:30.482048 9768 consensus_queue.cc:260] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 42709 } } peers { permanent_uuid: "2387d5bdca57450fb3df77183cc13423" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 39273 } } peers { permanent_uuid: "83289c6d98ce484786dfc77961e5884b" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45185 } }
I20260809 09:31:30.482607 9773 tablet_bootstrap.cc:654] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:30.482721 9265 heartbeater.cc:503] Master 127.9.102.62:38397 was elected leader, sending a full tablet report...
I20260809 09:31:30.483035 9768 ts_tablet_manager.cc:1428] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260809 09:31:30.483403 9773 log.cc:824] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:30.483650 9739 heartbeater.cc:503] Master 127.9.102.62:38397 was elected leader, sending a full tablet report...
I20260809 09:31:30.484247 9765 ts_tablet_manager.cc:1428] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260809 09:31:30.485175 9773 tablet_bootstrap.cc:492] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b: No bootstrap required, opened a new log
I20260809 09:31:30.485361 9773 ts_tablet_manager.cc:1397] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 09:31:30.488610 9773 raft_consensus.cc:348] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 42709 } } peers { permanent_uuid: "2387d5bdca57450fb3df77183cc13423" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 39273 } } peers { permanent_uuid: "83289c6d98ce484786dfc77961e5884b" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45185 } }
I20260809 09:31:30.488818 9773 raft_consensus.cc:374] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:30.488909 9773 raft_consensus.cc:729] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83289c6d98ce484786dfc77961e5884b, State: Initialized, Role: FOLLOWER
I20260809 09:31:30.489292 9773 consensus_queue.cc:260] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 42709 } } peers { permanent_uuid: "2387d5bdca57450fb3df77183cc13423" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 39273 } } peers { permanent_uuid: "83289c6d98ce484786dfc77961e5884b" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45185 } }
I20260809 09:31:30.489876 9499 heartbeater.cc:503] Master 127.9.102.62:38397 was elected leader, sending a full tablet report...
I20260809 09:31:30.490141 9773 ts_tablet_manager.cc:1428] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
W20260809 09:31:30.497089 9500 tablet.cc:2369] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:31:30.568288 9266 tablet.cc:2369] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:31:30.574481 9775 raft_consensus.cc:482] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:31:30.574628 9775 raft_consensus.cc:504] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 42709 } } peers { permanent_uuid: "2387d5bdca57450fb3df77183cc13423" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 39273 } } peers { permanent_uuid: "83289c6d98ce484786dfc77961e5884b" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45185 } }
I20260809 09:31:30.575673 9775 leader_election.cc:288] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2387d5bdca57450fb3df77183cc13423 (127.9.102.3:39273), 83289c6d98ce484786dfc77961e5884b (127.9.102.2:45185)
I20260809 09:31:30.579264 9267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.575881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:39273 (local address 127.9.102.1:51655)
0809 09:31:30.575996 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:30.576012 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:30.576122 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:30.576389 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:30.576394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:30.576413 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:30.576518 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.576526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.577501 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.577504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.578270 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.578279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.578907 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.578911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.578985 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.579004 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 09:31:30.579090 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 09:31:30.579154 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 09:31:30.579265 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.575998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:51655 (local address 127.9.102.3:39273)
0809 09:31:30.576144 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:30.576149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:30.576167 (+ 18us) server_negotiation.cc:407] Connection header received
0809 09:31:30.576224 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:30.576227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:30.576270 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:30.576348 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:30.576643 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.577386 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.578417 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.578784 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.578907 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.578961 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:30.579107 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 09:31:30.579172 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:31:30.579419 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.576257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:41257 (local address 127.9.102.2:45185)
0809 09:31:30.576371 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:30.576375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:30.576456 (+ 81us) server_negotiation.cc:407] Connection header received
0809 09:31:30.576547 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:30.576549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:30.576606 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:30.576679 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:30.576916 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.577548 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.578552 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.578906 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.579015 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.579064 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:30.579243 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 09:31:30.579324 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 09:31:30.579443 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.576163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:45185 (local address 127.9.102.1:41257)
0809 09:31:30.576383 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:30.576394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:30.576468 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:30.576699 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:30.576701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:30.576715 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:30.576807 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.576811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.577664 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.577667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.578418 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:30.578427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.579036 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.579039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:30.579132 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.579147 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 09:31:30.579226 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 09:31:30.579303 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":61,"threads_started":1}
I20260809 09:31:30.579959 9416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9a6ccd68ec470e8886555393f35270" candidate_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83289c6d98ce484786dfc77961e5884b" is_pre_election: true
I20260809 09:31:30.580067 9651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9a6ccd68ec470e8886555393f35270" candidate_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2387d5bdca57450fb3df77183cc13423" is_pre_election: true
I20260809 09:31:30.580262 9416 raft_consensus.cc:2454] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c6f567d7d948f8b1ab1739ff94c779 in term 0.
I20260809 09:31:30.580351 9651 raft_consensus.cc:2454] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2c6f567d7d948f8b1ab1739ff94c779 in term 0.
I20260809 09:31:30.580775 9066 leader_election.cc:302] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2387d5bdca57450fb3df77183cc13423, e2c6f567d7d948f8b1ab1739ff94c779; no voters:
I20260809 09:31:30.580965 9775 raft_consensus.cc:2781] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:31:30.581051 9775 raft_consensus.cc:482] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:31:30.581107 9775 raft_consensus.cc:3037] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:30.582028 9775 raft_consensus.cc:504] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 42709 } } peers { permanent_uuid: "2387d5bdca57450fb3df77183cc13423" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 39273 } } peers { permanent_uuid: "83289c6d98ce484786dfc77961e5884b" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45185 } }
I20260809 09:31:30.582305 9775 leader_election.cc:288] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [CANDIDATE]: Term 1 election: Requested vote from peers 2387d5bdca57450fb3df77183cc13423 (127.9.102.3:39273), 83289c6d98ce484786dfc77961e5884b (127.9.102.2:45185)
I20260809 09:31:30.582669 9651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9a6ccd68ec470e8886555393f35270" candidate_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2387d5bdca57450fb3df77183cc13423"
I20260809 09:31:30.582669 9416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f9a6ccd68ec470e8886555393f35270" candidate_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83289c6d98ce484786dfc77961e5884b"
I20260809 09:31:30.582777 9651 raft_consensus.cc:3037] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:30.582777 9416 raft_consensus.cc:3037] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:30.583886 9416 raft_consensus.cc:2454] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c6f567d7d948f8b1ab1739ff94c779 in term 1.
I20260809 09:31:30.583886 9651 raft_consensus.cc:2454] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2c6f567d7d948f8b1ab1739ff94c779 in term 1.
I20260809 09:31:30.584182 9066 leader_election.cc:302] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2387d5bdca57450fb3df77183cc13423, e2c6f567d7d948f8b1ab1739ff94c779; no voters:
I20260809 09:31:30.584323 9775 raft_consensus.cc:2781] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:30.584522 9775 raft_consensus.cc:686] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 1 LEADER]: Becoming Leader. State: Replica: e2c6f567d7d948f8b1ab1739ff94c779, State: Running, Role: LEADER
I20260809 09:31:30.584810 9775 consensus_queue.cc:237] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 42709 } } peers { permanent_uuid: "2387d5bdca57450fb3df77183cc13423" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 39273 } } peers { permanent_uuid: "83289c6d98ce484786dfc77961e5884b" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45185 } }
I20260809 09:31:30.587447 8921 catalog_manager.cc:5495] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2c6f567d7d948f8b1ab1739ff94c779 (127.9.102.1). New cstate: current_term: 1 leader_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c6f567d7d948f8b1ab1739ff94c779" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 42709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2387d5bdca57450fb3df77183cc13423" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 39273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83289c6d98ce484786dfc77961e5884b" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45185 } health_report { overall_health: UNKNOWN } } }
I20260809 09:31:30.611593 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.608833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.9.102.1:42709)
0809 09:31:30.608931 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:30.608934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:30.608945 (+ 11us) server_negotiation.cc:407] Connection header received
0809 09:31:30.609044 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:30.609046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:30.609087 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:30.609160 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:31:30.609347 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.609741 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.610479 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:30.610656 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:30.610720 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:30.610943 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:31:30.611271 (+ 328us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:31:30.611369 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:30.611466 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 09:31:30.611491 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260809 09:31:30.616415 9416 raft_consensus.cc:1261] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b [term 1 FOLLOWER]: Refusing update from remote peer e2c6f567d7d948f8b1ab1739ff94c779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:31:30.616430 9651 raft_consensus.cc:1261] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423 [term 1 FOLLOWER]: Refusing update from remote peer e2c6f567d7d948f8b1ab1739ff94c779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:31:30.617098 9775 consensus_queue.cc:1035] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83289c6d98ce484786dfc77961e5884b" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 45185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:31:30.617269 9782 consensus_queue.cc:1035] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2387d5bdca57450fb3df77183cc13423" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 39273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:31:30.623880 9787 mvcc.cc:205] Tried to move back new op lower bound from 7316553279958663168 to 7316553279837495296. Current Snapshot: MvccSnapshot[applied={T|T < 7316553279958663168}]
I20260809 09:31:30.625833 9788 mvcc.cc:205] Tried to move back new op lower bound from 7316553279958663168 to 7316553279837495296. Current Snapshot: MvccSnapshot[applied={T|T < 7316553279958663168}]
I20260809 09:31:30.630995 8922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:31:30.631119 8922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:31:30.633294 8922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 897ed7fa570f467590470f0b997d9ebb: Sending DeleteTablet for 3 replicas of tablet 8f9a6ccd68ec470e8886555393f35270
I20260809 09:31:30.634059 9154 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f9a6ccd68ec470e8886555393f35270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:30 UTC) from {username='slave'} at 127.0.0.1:37420
I20260809 09:31:30.634267 9614 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f9a6ccd68ec470e8886555393f35270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:30 UTC) from {username='slave'} at 127.0.0.1:48180
I20260809 09:31:30.634303 9381 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f9a6ccd68ec470e8886555393f35270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:30 UTC) from {username='slave'} at 127.0.0.1:37762
I20260809 09:31:30.634495 9799 tablet_replica.cc:330] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779: stopping tablet replica
I20260809 09:31:30.634779 9799 raft_consensus.cc:2229] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:31:30.635103 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 9037
I20260809 09:31:30.635146 9799 raft_consensus.cc:2258] T 8f9a6ccd68ec470e8886555393f35270 P e2c6f567d7d948f8b1ab1739ff94c779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:30.638142 9800 tablet_replica.cc:330] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423: stopping tablet replica
I20260809 09:31:30.638391 9800 raft_consensus.cc:2229] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:31:30.638650 9800 raft_consensus.cc:2258] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:30.639027 9801 tablet_replica.cc:330] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b: stopping tablet replica
I20260809 09:31:30.639256 9801 raft_consensus.cc:2229] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:31:30.639452 9801 raft_consensus.cc:2258] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:30.642120 9801 ts_tablet_manager.cc:1905] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:31:30.642120 9800 ts_tablet_manager.cc:1905] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:31:30.644543 9800 ts_tablet_manager.cc:1918] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:31:30.644543 9801 ts_tablet_manager.cc:1918] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:31:30.644640 9800 log.cc:1196] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/8f9a6ccd68ec470e8886555393f35270
I20260809 09:31:30.644640 9801 log.cc:1196] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/8f9a6ccd68ec470e8886555393f35270
I20260809 09:31:30.644958 9800 ts_tablet_manager.cc:1939] T 8f9a6ccd68ec470e8886555393f35270 P 2387d5bdca57450fb3df77183cc13423: Deleting consensus metadata
I20260809 09:31:30.644958 9801 ts_tablet_manager.cc:1939] T 8f9a6ccd68ec470e8886555393f35270 P 83289c6d98ce484786dfc77961e5884b: Deleting consensus metadata
I20260809 09:31:30.646155 8890 catalog_manager.cc:4841] TS 2387d5bdca57450fb3df77183cc13423 (127.9.102.3:39273): tablet 8f9a6ccd68ec470e8886555393f35270 (table test-table [id=310b16afdf37442ab2bfc4810d334709]) successfully deleted
I20260809 09:31:30.646368 8890 catalog_manager.cc:4841] TS 83289c6d98ce484786dfc77961e5884b (127.9.102.2:45185): tablet 8f9a6ccd68ec470e8886555393f35270 (table test-table [id=310b16afdf37442ab2bfc4810d334709]) successfully deleted
W20260809 09:31:30.646621 8893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:42709 (error 108)
I20260809 09:31:30.646909 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 9269
I20260809 09:31:30.654160 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:30.653844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:42709 (local address 127.0.0.1:37446)
0809 09:31:30.654030 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:30.654093 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:42709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260809 09:31:30.654313 8893 catalog_manager.cc:4584] TS e2c6f567d7d948f8b1ab1739ff94c779 (127.9.102.1:42709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f9a6ccd68ec470e8886555393f35270: Network error: Client connection negotiation failed: client connection to 127.9.102.1:42709: connect: Connection refused (error 111)
I20260809 09:31:30.660041 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 9503
I20260809 09:31:30.679809 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 8854
2026-08-09T09:31:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1026 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:30 dist-test-slave-7vjv krb5kdc[9805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:30 dist-test-slave-7vjv krb5kdc[9805](info): set up 2 sockets
Aug 09 09:31:30 dist-test-slave-7vjv krb5kdc[9805](info): commencing operation
krb5kdc: starting...
W20260809 09:31:31.814096 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:31 dist-test-slave-7vjv krb5kdc[9805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:31:31.923182 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:45425
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:45833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:45425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:32.020815 9830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:32.021049 9830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:32.021111 9830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:32.024524 9830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:32.024595 9830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:32.024631 9830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:32.024684 9830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:32.024720 9830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:32.028927 9830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:45833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:45425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:45425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:32.029712 9830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:32.030520 9830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:32.038169 9836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:32.038201 9840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:32.038210 9835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:32.038255 9830 server_base.cc:1034] running on GCE node
I20260809 09:31:32.038949 9830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:32.039920 9830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:32.041105 9830 hybrid_clock.cc:648] HybridClock initialized: now 1786267892041080 us; error 41 us; skew 500 ppm
Aug 09 09:31:32 dist-test-slave-7vjv krb5kdc[9805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267892, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:32.043638 9830 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:31:32.044456 9830 webserver.cc:460] Webserver started at http://127.9.102.62:38219/ using document root <none> and password file <none>
I20260809 09:31:32.044963 9830 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:32.045050 9830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:32.045258 9830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:32.047003 9830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "a3cbf47f563e4e178168d6551c657543"
format_stamp: "Formatted at 2026-08-09 09:31:32 on dist-test-slave-7vjv"
server_key: "500383d859f07fad8ebd02bd50d46068"
server_key_iv: "4059bbfafe9f8e775cf32812cbe26fff"
server_key_version: "encryptionkey@0"
I20260809 09:31:32.047509 9830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "a3cbf47f563e4e178168d6551c657543"
format_stamp: "Formatted at 2026-08-09 09:31:32 on dist-test-slave-7vjv"
server_key: "500383d859f07fad8ebd02bd50d46068"
server_key_iv: "4059bbfafe9f8e775cf32812cbe26fff"
server_key_version: "encryptionkey@0"
I20260809 09:31:32.050756 9830 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 09:31:32.052947 9851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:32.053972 9830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:31:32.054104 9830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "a3cbf47f563e4e178168d6551c657543"
format_stamp: "Formatted at 2026-08-09 09:31:32 on dist-test-slave-7vjv"
server_key: "500383d859f07fad8ebd02bd50d46068"
server_key_iv: "4059bbfafe9f8e775cf32812cbe26fff"
server_key_version: "encryptionkey@0"
I20260809 09:31:32.054203 9830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:32.085585 9830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:32.088043 9830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:32.088266 9830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:32.094738 9830 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:45425
I20260809 09:31:32.094764 9957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:45425 every 8 connection(s)
I20260809 09:31:32.095597 9830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:32.098094 9958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:32.099144 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 9830
I20260809 09:31:32.099272 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:32.099560 9624 external_mini_cluster.cc:1428] Setting key 7a29a9f273da5587a49728977afe4a42
I20260809 09:31:32.103708 9958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543: Bootstrap starting.
I20260809 09:31:32.105640 9958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:32.106408 9958 log.cc:824] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543: Log is configured to *not* fsync() on all Append() calls
Aug 09 09:31:32 dist-test-slave-7vjv krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:32.108224 9958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543: No bootstrap required, opened a new log
I20260809 09:31:32.112396 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.100927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.9.102.62:45425)
0809 09:31:32.101350 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:32.101363 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:32.101395 (+ 32us) server_negotiation.cc:407] Connection header received
0809 09:31:32.101946 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:32.101960 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:32.102249 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:32.102542 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:32.103281 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.104192 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.105005 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.105423 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.105580 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.107883 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:32.107890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:32.107905 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:32.107931 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:32.109718 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:32.110104 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:32.110109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:32.110113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:32.110253 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:32.110496 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:32.110500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:32.110503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:32.110936 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:32.111123 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:32.111283 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 09:31:32.111558 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":140,"threads_started":1}
I20260809 09:31:32.112347 9958 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cbf47f563e4e178168d6551c657543" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45425 } }
I20260809 09:31:32.112562 9958 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:32.112618 9958 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cbf47f563e4e178168d6551c657543, State: Initialized, Role: FOLLOWER
I20260809 09:31:32.113075 9958 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cbf47f563e4e178168d6551c657543" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45425 } }
I20260809 09:31:32.113214 9958 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:32.113276 9958 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:32.113392 9958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:32.114163 9958 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cbf47f563e4e178168d6551c657543" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45425 } }
I20260809 09:31:32.114503 9958 leader_election.cc:302] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3cbf47f563e4e178168d6551c657543; no voters:
I20260809 09:31:32.114785 9958 leader_election.cc:288] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:32.114924 9964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:32.115129 9964 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [term 1 LEADER]: Becoming Leader. State: Replica: a3cbf47f563e4e178168d6551c657543, State: Running, Role: LEADER
I20260809 09:31:32.115482 9964 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cbf47f563e4e178168d6551c657543" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45425 } }
I20260809 09:31:32.115940 9958 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:32.117087 9968 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3cbf47f563e4e178168d6551c657543. Latest consensus state: current_term: 1 leader_uuid: "a3cbf47f563e4e178168d6551c657543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cbf47f563e4e178168d6551c657543" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45425 } } }
I20260809 09:31:32.117091 9967 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3cbf47f563e4e178168d6551c657543" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3cbf47f563e4e178168d6551c657543" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45425 } } }
I20260809 09:31:32.117262 9967 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:32.117560 9968 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:32.117606 9978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:32.120671 9978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:32.125494 9978 catalog_manager.cc:1347] Generated new cluster ID: 0834a1c5180f4df097465357c0e28e15
I20260809 09:31:32.125551 9978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:32.134892 9978 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:32.135864 9978 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:32.141124 9978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543: Generated new TSK 0
I20260809 09:31:32.141697 9978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
Entry for principal kudu/127.9.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab.
I20260809 09:31:32.172868 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:45425
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:45833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:32.271381 9997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:32.271723 9997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:32.271801 9997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:32.275588 9997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:32.275666 9997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:32.275764 9997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:31:32.280511 9997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:45833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.1.keytab
--principal=kudu/127.9.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:45425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:32.281342 9997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:32.282191 9997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:32.290488 10007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:32.290488 10005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:32.290578 9997 server_base.cc:1034] running on GCE node
W20260809 09:31:32.290578 10002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:32.291054 9997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:32.291576 9997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:32.292724 9997 hybrid_clock.cc:648] HybridClock initialized: now 1786267892292707 us; error 27 us; skew 500 ppm
Aug 09 09:31:32 dist-test-slave-7vjv krb5kdc[9805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267892, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:32.295212 9997 init.cc:377] Logged in from keytab as kudu/127.9.102.1@KRBTEST.COM (short username kudu)
I20260809 09:31:32.295954 9997 webserver.cc:460] Webserver started at http://127.9.102.1:43257/ using document root <none> and password file <none>
I20260809 09:31:32.296509 9997 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:32.296578 9997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:32.296816 9997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:32.298511 9997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "ba8fd7489dc34911b207d2513067b009"
format_stamp: "Formatted at 2026-08-09 09:31:32 on dist-test-slave-7vjv"
server_key: "4b839ead5de562bf4a9abf42d7af042b"
server_key_iv: "552ed0177b3470fc87853fe9d501c9de"
server_key_version: "encryptionkey@0"
I20260809 09:31:32.299022 9997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "ba8fd7489dc34911b207d2513067b009"
format_stamp: "Formatted at 2026-08-09 09:31:32 on dist-test-slave-7vjv"
server_key: "4b839ead5de562bf4a9abf42d7af042b"
server_key_iv: "552ed0177b3470fc87853fe9d501c9de"
server_key_version: "encryptionkey@0"
I20260809 09:31:32.302331 9997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:31:32.304440 10020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:32.305442 9997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:32.305563 9997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "ba8fd7489dc34911b207d2513067b009"
format_stamp: "Formatted at 2026-08-09 09:31:32 on dist-test-slave-7vjv"
server_key: "4b839ead5de562bf4a9abf42d7af042b"
server_key_iv: "552ed0177b3470fc87853fe9d501c9de"
server_key_version: "encryptionkey@0"
I20260809 09:31:32.305653 9997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:32.314826 9997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:32.317631 9997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:32.317803 9997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:32.318282 9997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:32.318970 9997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:32.319012 9997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:32.319075 9997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:32.319104 9997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:32.327172 9997 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:40067
I20260809 09:31:32.327193 10196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:40067 every 8 connection(s)
I20260809 09:31:32.327876 9997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:31:32.328076 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 9997
I20260809 09:31:32.328244 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:31:32.328523 9624 external_mini_cluster.cc:1428] Setting key 61a9b48777cf489560b09568fd852e01
Aug 09 09:31:32 dist-test-slave-7vjv krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267892, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.1@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.102.2@KRBTEST.COM; defaulting to no policy
I20260809 09:31:32.343614 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.329511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:53053 (local address 127.9.102.62:45425)
0809 09:31:32.329644 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:32.329648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:32.330431 (+ 783us) server_negotiation.cc:407] Connection header received
0809 09:31:32.331756 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:32.331761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:32.331820 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:32.331910 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:32.333395 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.334146 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.335273 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.335612 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.335727 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.338773 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:32.338776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:32.338779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:32.338808 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:32.340391 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:32.340990 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:32.341004 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:32.341006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:32.341098 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:32.341447 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:32.341452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:32.341454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:32.341672 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:32.341756 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:32.343339 (+ 1583us) server_negotiation.cc:299] Negotiation successful
0809 09:31:32.343457 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 09:31:32.344547 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.329856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:45425 (local address 127.9.102.1:53053)
0809 09:31:32.330279 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:32.330324 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:32.331571 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:32.332068 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:32.332079 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:32.332573 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:32.333232 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:32.333256 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.334273 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.334276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:32.335141 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:32.335153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.335910 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.335915 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:32.336148 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.336726 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:31:32.336745 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:32.338555 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:32.340521 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:32.340539 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:32.340553 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:32.340904 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:32.341182 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:32.341186 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:32.341190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:32.341353 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:32.342789 (+ 1436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:32.342794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:32.343081 (+ 287us) client_negotiation.cc:769] Sending connection context
0809 09:31:32.343281 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 09:31:32.343562 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.2@KRBTEST.COM" created.
I20260809 09:31:32.345952 10197 heartbeater.cc:347] Connected to a master server at 127.9.102.62:45425
I20260809 09:31:32.346210 10197 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:32.346817 10197 heartbeater.cc:511] Master 127.9.102.62:45425 requested a full tablet report, sending...
I20260809 09:31:32.348382 9894 ts_manager.cc:194] Registered new tserver with Master: ba8fd7489dc34911b207d2513067b009 (127.9.102.1:40067)
I20260809 09:31:32.349426 9894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.1@KRBTEST.COM'} at 127.9.102.1:53053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
Entry for principal kudu/127.9.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab.
I20260809 09:31:32.362279 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:45425
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:45833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:32.467430 10202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:32.467644 10202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:32.467681 10202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:32.471293 10202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:32.471339 10202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:32.471402 10202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:31:32.475919 10202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:45833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.2.keytab
--principal=kudu/127.9.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:45425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:32.476722 10202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:32.477545 10202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:32.485942 10213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:32.485955 10215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:32.486001 10202 server_base.cc:1034] running on GCE node
W20260809 09:31:32.485957 10211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:32.486492 10202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:32.487077 10202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:32.488240 10202 hybrid_clock.cc:648] HybridClock initialized: now 1786267892488222 us; error 32 us; skew 500 ppm
Aug 09 09:31:32 dist-test-slave-7vjv krb5kdc[9805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267892, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:32.490929 10202 init.cc:377] Logged in from keytab as kudu/127.9.102.2@KRBTEST.COM (short username kudu)
I20260809 09:31:32.491631 10202 webserver.cc:460] Webserver started at http://127.9.102.2:37757/ using document root <none> and password file <none>
I20260809 09:31:32.492208 10202 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:32.492271 10202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:32.492487 10202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:32.494222 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "6acde597072043feb32cb932fd0d4a72"
format_stamp: "Formatted at 2026-08-09 09:31:32 on dist-test-slave-7vjv"
server_key: "7ed2bfc216da9fb5520e20a2debffde2"
server_key_iv: "ef4d040f5b0fe402d6401721fd820045"
server_key_version: "encryptionkey@0"
I20260809 09:31:32.494710 10202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "6acde597072043feb32cb932fd0d4a72"
format_stamp: "Formatted at 2026-08-09 09:31:32 on dist-test-slave-7vjv"
server_key: "7ed2bfc216da9fb5520e20a2debffde2"
server_key_iv: "ef4d040f5b0fe402d6401721fd820045"
server_key_version: "encryptionkey@0"
I20260809 09:31:32.498162 10202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:31:32.500344 10232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:32.501420 10202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:32.501528 10202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "6acde597072043feb32cb932fd0d4a72"
format_stamp: "Formatted at 2026-08-09 09:31:32 on dist-test-slave-7vjv"
server_key: "7ed2bfc216da9fb5520e20a2debffde2"
server_key_iv: "ef4d040f5b0fe402d6401721fd820045"
server_key_version: "encryptionkey@0"
I20260809 09:31:32.501616 10202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:32.516430 10202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:32.519029 10202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:32.519220 10202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:32.519716 10202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:32.520530 10202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:32.520572 10202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:32.520605 10202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:32.520619 10202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:32.528285 10202 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:43019
I20260809 09:31:32.528360 10427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:43019 every 8 connection(s)
I20260809 09:31:32.529043 10202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
Aug 09 09:31:32 dist-test-slave-7vjv krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267892, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.2@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:32.538319 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 10202
I20260809 09:31:32.538411 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:31:32.538722 9624 external_mini_cluster.cc:1428] Setting key 54f895e83cf0b59f78240a88f495d7c8
I20260809 09:31:32.540364 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.530585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:50057 (local address 127.9.102.62:45425)
0809 09:31:32.530707 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:32.530710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:32.531380 (+ 670us) server_negotiation.cc:407] Connection header received
0809 09:31:32.532488 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:32.532491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:32.532533 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:32.532597 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:32.533505 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.533956 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.534921 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.535100 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.535165 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.537305 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:32.537308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:32.537310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:32.537331 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:32.538397 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:32.538914 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:32.538916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:32.538918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:32.538991 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:32.539341 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:32.539343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:32.539344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:32.539487 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:32.539599 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:32.540113 (+ 514us) server_negotiation.cc:299] Negotiation successful
0809 09:31:32.540229 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 09:31:32.541038 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.530864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:45425 (local address 127.9.102.2:50057)
0809 09:31:32.531255 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:32.531288 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:32.532320 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:32.532741 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:32.532746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:32.533024 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:32.533399 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:32.533410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.534050 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.534052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:32.534820 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:32.534828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.535211 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.535213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:32.535330 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.535824 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:31:32.535838 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:32.537170 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:32.538546 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:32.538552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:32.538563 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:32.538812 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:32.539099 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:32.539102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:32.539103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:32.539251 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:32.539604 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:32.539611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:32.539844 (+ 233us) client_negotiation.cc:769] Sending connection context
0809 09:31:32.540034 (+ 190us) client_negotiation.cc:240] Negotiation successful
0809 09:31:32.540306 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":123,"threads_started":1}
I20260809 09:31:32.542048 10428 heartbeater.cc:347] Connected to a master server at 127.9.102.62:45425
I20260809 09:31:32.542263 10428 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:32.542658 10428 heartbeater.cc:511] Master 127.9.102.62:45425 requested a full tablet report, sending...
I20260809 09:31:32.543555 9894 ts_manager.cc:194] Registered new tserver with Master: 6acde597072043feb32cb932fd0d4a72 (127.9.102.2:43019)
I20260809 09:31:32.544032 9894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.2@KRBTEST.COM'} at 127.9.102.2:50057
WARNING: no policy specified for kudu/127.9.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
Entry for principal kudu/127.9.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab.
I20260809 09:31:32.563481 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:45425
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:45833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:32.667114 10436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:32.667323 10436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:32.667361 10436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:32.671523 10436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:32.671595 10436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:32.671675 10436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:31:32.680809 10436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:45833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.3.keytab
--principal=kudu/127.9.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:45425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:32.681946 10436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:32.683120 10436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:32.695875 10444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:32.695879 10446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:32.695888 10449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:32.696604 10436 server_base.cc:1034] running on GCE node
I20260809 09:31:32.697005 10436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:32.697553 10436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:32.698736 10436 hybrid_clock.cc:648] HybridClock initialized: now 1786267892698708 us; error 48 us; skew 500 ppm
Aug 09 09:31:32 dist-test-slave-7vjv krb5kdc[9805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267892, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:32.702513 10436 init.cc:377] Logged in from keytab as kudu/127.9.102.3@KRBTEST.COM (short username kudu)
I20260809 09:31:32.703420 10436 webserver.cc:460] Webserver started at http://127.9.102.3:41921/ using document root <none> and password file <none>
I20260809 09:31:32.703926 10436 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:32.704039 10436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:32.704360 10436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:32.706873 10436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "c8a74cf1b6bc48fa85999359b0311afc"
format_stamp: "Formatted at 2026-08-09 09:31:32 on dist-test-slave-7vjv"
server_key: "df4a6714dde58baded08140feeff6372"
server_key_iv: "68c891bf991de2f42f30654f29cdddfa"
server_key_version: "encryptionkey@0"
I20260809 09:31:32.707561 10436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "c8a74cf1b6bc48fa85999359b0311afc"
format_stamp: "Formatted at 2026-08-09 09:31:32 on dist-test-slave-7vjv"
server_key: "df4a6714dde58baded08140feeff6372"
server_key_iv: "68c891bf991de2f42f30654f29cdddfa"
server_key_version: "encryptionkey@0"
I20260809 09:31:32.712513 10436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 09:31:32.715508 10466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:32.716698 10436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:31:32.716790 10436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "c8a74cf1b6bc48fa85999359b0311afc"
format_stamp: "Formatted at 2026-08-09 09:31:32 on dist-test-slave-7vjv"
server_key: "df4a6714dde58baded08140feeff6372"
server_key_iv: "68c891bf991de2f42f30654f29cdddfa"
server_key_version: "encryptionkey@0"
I20260809 09:31:32.716923 10436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:32.735277 10436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:32.737753 10436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:32.737941 10436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:32.738417 10436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:32.739113 10436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:32.739189 10436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:32.739269 10436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:32.739316 10436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:32.747223 10436 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:38645
I20260809 09:31:32.747293 10656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:38645 every 8 connection(s)
I20260809 09:31:32.747973 10436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:31:32.751718 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 10436
I20260809 09:31:32.751839 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:31:32.752084 9624 external_mini_cluster.cc:1428] Setting key f5604d3ef7cfa187c7223e25c4d54958
Aug 09 09:31:32 dist-test-slave-7vjv krb5kdc[9805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267892, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.3@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:32.760799 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.749861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:36273 (local address 127.9.102.62:45425)
0809 09:31:32.749995 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:32.750000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:32.750774 (+ 774us) server_negotiation.cc:407] Connection header received
0809 09:31:32.751914 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:32.751917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:32.751972 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:32.752014 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:32.753420 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.754070 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.755058 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.755251 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.755330 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.757624 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:32.757632 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:32.757634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:32.757656 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:32.758896 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:32.759395 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:32.759399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:32.759402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:32.759478 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:32.759798 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:32.759802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:32.759803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:32.759980 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:32.760084 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:32.760537 (+ 453us) server_negotiation.cc:299] Negotiation successful
0809 09:31:32.760648 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 09:31:32.761399 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.750224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:45425 (local address 127.9.102.3:36273)
0809 09:31:32.750612 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:32.750667 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:32.751676 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:32.752357 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:32.752362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:32.752754 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:32.753255 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:32.753273 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.754187 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.754190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:32.754948 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:32.754957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.755383 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.755386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:32.755508 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.756018 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:31:32.756033 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:32.757469 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:32.759013 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:32.759018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:32.759030 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:32.759267 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:32.759566 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:31:32.759570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:31:32.759571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:31:32.759704 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:31:32.760084 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:32.760111 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:32.760316 (+ 205us) client_negotiation.cc:769] Sending connection context
0809 09:31:32.760492 (+ 176us) client_negotiation.cc:240] Negotiation successful
0809 09:31:32.760752 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":114,"threads_started":1}
I20260809 09:31:32.762303 10657 heartbeater.cc:347] Connected to a master server at 127.9.102.62:45425
I20260809 09:31:32.762526 10657 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:32.762967 10657 heartbeater.cc:511] Master 127.9.102.62:45425 requested a full tablet report, sending...
I20260809 09:31:32.763862 9894 ts_manager.cc:194] Registered new tserver with Master: c8a74cf1b6bc48fa85999359b0311afc (127.9.102.3:38645)
I20260809 09:31:32.764496 9894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.102.3@KRBTEST.COM'} at 127.9.102.3:36273
I20260809 09:31:32.765822 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:31:32.774163 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.767699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.9.102.62:45425)
0809 09:31:32.767918 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:32.767922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:32.768021 (+ 99us) server_negotiation.cc:407] Connection header received
0809 09:31:32.768155 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:32.768162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:32.768196 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:32.768259 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:32.768885 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.769344 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.770070 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.770260 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.770345 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.771622 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:32.771624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:32.771626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:32.771641 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:32.772824 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:32.773236 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:32.773238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:32.773239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:32.773313 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:32.773533 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:32.773535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:32.773536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:32.773704 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:32.773795 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:32.773985 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 09:31:32.774072 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260809 09:31:32.776772 9883 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:31:32.779351 9883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:31:32.792443 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.786897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:40067 (local address 127.0.0.1:49592)
0809 09:31:32.787810 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:32.787823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:32.787897 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:32.788639 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:32.788643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:32.788675 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:32.788778 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:32.788782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.789839 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.789843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:32.790926 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:32.790937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.791924 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.791927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:32.792025 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.792056 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 09:31:32.792184 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 09:31:32.792302 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":2,"spinlock_wait_cycles":9856,"thread_start_us":80,"threads_started":1}
I20260809 09:31:32.792457 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.787420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.9.102.1:40067)
0809 09:31:32.787837 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:32.787848 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:32.787890 (+ 42us) server_negotiation.cc:407] Connection header received
0809 09:31:32.788034 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:32.788039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:32.788269 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:32.788432 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:32.788930 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.789722 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.791067 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.791794 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.791891 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.792004 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:32.792211 (+ 207us) server_negotiation.cc:299] Negotiation successful
0809 09:31:32.792320 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":141,"threads_started":1}
I20260809 09:31:32.792665 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.786717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:43019 (local address 127.0.0.1:57848)
0809 09:31:32.787383 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:32.787424 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:32.787565 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:32.788643 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:32.788647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:32.788677 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:32.788842 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:32.788850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.790237 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.790240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:32.791234 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:32.791245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.792164 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.792169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:32.792274 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.792293 (+ 19us) client_negotiation.cc:769] Sending connection context
0809 09:31:32.792363 (+ 70us) client_negotiation.cc:240] Negotiation successful
0809 09:31:32.792411 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":23,"spinlock_wait_cycles":2688,"thread_start_us":67,"threads_started":1}
I20260809 09:31:32.793144 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.786889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.9.102.2:43019)
0809 09:31:32.787339 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:32.787351 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:32.787522 (+ 171us) server_negotiation.cc:407] Connection header received
0809 09:31:32.787716 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:32.787723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:32.788320 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:32.788480 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:32.788978 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.790106 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.791333 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.791816 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.791932 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.792065 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:32.792953 (+ 888us) server_negotiation.cc:299] Negotiation successful
0809 09:31:32.793037 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":163,"threads_started":1}
I20260809 09:31:32.793982 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.787424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.9.102.3:38645)
0809 09:31:32.788447 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:32.788459 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:32.788480 (+ 21us) server_negotiation.cc:407] Connection header received
0809 09:31:32.788553 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:32.788558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:32.788710 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:32.788856 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:32.789113 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.790122 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.792499 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.793404 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.793518 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.793642 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:32.793776 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 09:31:32.793853 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":135,"threads_started":1}
I20260809 09:31:32.794193 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:32.787124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:38645 (local address 127.0.0.1:47096)
0809 09:31:32.787926 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:32.787962 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:32.788048 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:32.788850 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:32.788855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:32.788876 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:32.788981 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:32.788987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.791353 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.791355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:32.792370 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:32.792380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:32.793537 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:32.793541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:32.793636 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:32.793658 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 09:31:32.793739 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 09:31:32.793937 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":70,"threads_started":1}
I20260809 09:31:32.795365 10092 tablet_service.cc:1467] Processing CreateTablet for tablet ffc06a07548a43618c16262ceb9ba193 (DEFAULT_TABLE table=test-table [id=6a14c485f73b46ccb7101eb59eabeb9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:32.795506 10307 tablet_service.cc:1467] Processing CreateTablet for tablet ffc06a07548a43618c16262ceb9ba193 (DEFAULT_TABLE table=test-table [id=6a14c485f73b46ccb7101eb59eabeb9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:32.796334 10092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc06a07548a43618c16262ceb9ba193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:32.796455 10307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc06a07548a43618c16262ceb9ba193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:32.796512 10541 tablet_service.cc:1467] Processing CreateTablet for tablet ffc06a07548a43618c16262ceb9ba193 (DEFAULT_TABLE table=test-table [id=6a14c485f73b46ccb7101eb59eabeb9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:32.797317 10541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc06a07548a43618c16262ceb9ba193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:32.802052 10686 tablet_bootstrap.cc:492] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009: Bootstrap starting.
I20260809 09:31:32.803269 10687 tablet_bootstrap.cc:492] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72: Bootstrap starting.
I20260809 09:31:32.803885 10691 tablet_bootstrap.cc:492] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc: Bootstrap starting.
I20260809 09:31:32.804411 10686 tablet_bootstrap.cc:654] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:32.805648 10687 tablet_bootstrap.cc:654] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:32.805944 10691 tablet_bootstrap.cc:654] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:32.806036 10686 log.cc:824] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:32.806437 10687 log.cc:824] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:32.807031 10691 log.cc:824] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:32.807945 10686 tablet_bootstrap.cc:492] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009: No bootstrap required, opened a new log
I20260809 09:31:32.808169 10686 ts_tablet_manager.cc:1397] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260809 09:31:32.808290 10687 tablet_bootstrap.cc:492] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72: No bootstrap required, opened a new log
I20260809 09:31:32.808459 10687 ts_tablet_manager.cc:1397] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:31:32.808846 10691 tablet_bootstrap.cc:492] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc: No bootstrap required, opened a new log
I20260809 09:31:32.809038 10691 ts_tablet_manager.cc:1397] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:31:32.811969 10691 raft_consensus.cc:348] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6acde597072043feb32cb932fd0d4a72" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 43019 } } peers { permanent_uuid: "ba8fd7489dc34911b207d2513067b009" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 40067 } } peers { permanent_uuid: "c8a74cf1b6bc48fa85999359b0311afc" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38645 } }
I20260809 09:31:32.812230 10691 raft_consensus.cc:374] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:32.812063 10686 raft_consensus.cc:348] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6acde597072043feb32cb932fd0d4a72" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 43019 } } peers { permanent_uuid: "ba8fd7489dc34911b207d2513067b009" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 40067 } } peers { permanent_uuid: "c8a74cf1b6bc48fa85999359b0311afc" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38645 } }
I20260809 09:31:32.812065 10687 raft_consensus.cc:348] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6acde597072043feb32cb932fd0d4a72" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 43019 } } peers { permanent_uuid: "ba8fd7489dc34911b207d2513067b009" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 40067 } } peers { permanent_uuid: "c8a74cf1b6bc48fa85999359b0311afc" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38645 } }
I20260809 09:31:32.812284 10691 raft_consensus.cc:729] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a74cf1b6bc48fa85999359b0311afc, State: Initialized, Role: FOLLOWER
I20260809 09:31:32.812309 10686 raft_consensus.cc:374] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:32.812309 10687 raft_consensus.cc:374] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:32.812357 10686 raft_consensus.cc:729] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8fd7489dc34911b207d2513067b009, State: Initialized, Role: FOLLOWER
I20260809 09:31:32.812366 10687 raft_consensus.cc:729] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6acde597072043feb32cb932fd0d4a72, State: Initialized, Role: FOLLOWER
I20260809 09:31:32.812683 10691 consensus_queue.cc:260] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6acde597072043feb32cb932fd0d4a72" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 43019 } } peers { permanent_uuid: "ba8fd7489dc34911b207d2513067b009" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 40067 } } peers { permanent_uuid: "c8a74cf1b6bc48fa85999359b0311afc" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38645 } }
I20260809 09:31:32.812753 10686 consensus_queue.cc:260] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6acde597072043feb32cb932fd0d4a72" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 43019 } } peers { permanent_uuid: "ba8fd7489dc34911b207d2513067b009" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 40067 } } peers { permanent_uuid: "c8a74cf1b6bc48fa85999359b0311afc" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38645 } }
I20260809 09:31:32.812753 10687 consensus_queue.cc:260] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6acde597072043feb32cb932fd0d4a72" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 43019 } } peers { permanent_uuid: "ba8fd7489dc34911b207d2513067b009" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 40067 } } peers { permanent_uuid: "c8a74cf1b6bc48fa85999359b0311afc" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38645 } }
I20260809 09:31:32.813231 10657 heartbeater.cc:503] Master 127.9.102.62:45425 was elected leader, sending a full tablet report...
I20260809 09:31:32.813501 10691 ts_tablet_manager.cc:1428] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260809 09:31:32.813572 10687 ts_tablet_manager.cc:1428] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260809 09:31:32.813870 10428 heartbeater.cc:503] Master 127.9.102.62:45425 was elected leader, sending a full tablet report...
I20260809 09:31:32.814553 10686 ts_tablet_manager.cc:1428] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260809 09:31:32.814980 10197 heartbeater.cc:503] Master 127.9.102.62:45425 was elected leader, sending a full tablet report...
W20260809 09:31:32.828956 10198 tablet.cc:2369] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:31:32.999017 10658 tablet.cc:2369] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 09:31:33.030174 10429 tablet.cc:2369] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:31:33.070521 10700 raft_consensus.cc:482] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:31:33.070683 10700 raft_consensus.cc:504] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6acde597072043feb32cb932fd0d4a72" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 43019 } } peers { permanent_uuid: "ba8fd7489dc34911b207d2513067b009" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 40067 } } peers { permanent_uuid: "c8a74cf1b6bc48fa85999359b0311afc" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38645 } }
I20260809 09:31:33.071730 10700 leader_election.cc:288] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba8fd7489dc34911b207d2513067b009 (127.9.102.1:40067), c8a74cf1b6bc48fa85999359b0311afc (127.9.102.3:38645)
I20260809 09:31:33.075628 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.072122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:39603 (local address 127.9.102.3:38645)
0809 09:31:33.072223 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:33.072228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:33.072346 (+ 118us) server_negotiation.cc:407] Connection header received
0809 09:31:33.072505 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:33.072507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:33.072567 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:33.072635 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:33.072969 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.073481 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.074526 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.075107 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.075220 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.075274 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:33.075444 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 09:31:33.075507 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 09:31:33.075662 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.071988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:39465 (local address 127.9.102.1:40067)
0809 09:31:33.072140 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:33.072144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:33.072300 (+ 156us) server_negotiation.cc:407] Connection header received
0809 09:31:33.072465 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:33.072469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:33.072511 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:33.072591 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:33.072846 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.073311 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.074493 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.075054 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.075138 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.075203 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:33.075466 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 09:31:33.075541 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 09:31:33.075670 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.071972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:38645 (local address 127.9.102.2:39603)
0809 09:31:33.072257 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:33.072267 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:33.072381 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:33.072720 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:33.072724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:33.072757 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:33.072856 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.072860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.073589 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.073594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:33.074340 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.074352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.075224 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.075227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:33.075322 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.075343 (+ 21us) client_negotiation.cc:769] Sending connection context
0809 09:31:33.075428 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 09:31:33.075496 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":24,"thread_start_us":113,"threads_started":1}
I20260809 09:31:33.075681 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.071896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:40067 (local address 127.9.102.2:39465)
0809 09:31:33.072202 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:33.072215 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:33.072330 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:33.072603 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:33.072606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:33.072621 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:33.072737 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.072742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.073423 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.073429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:33.074326 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.074335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.075277 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.075279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:33.075354 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.075368 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 09:31:33.075446 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 09:31:33.075512 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20260809 09:31:33.076339 10574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc06a07548a43618c16262ceb9ba193" candidate_uuid: "6acde597072043feb32cb932fd0d4a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a74cf1b6bc48fa85999359b0311afc" is_pre_election: true
I20260809 09:31:33.076442 10128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc06a07548a43618c16262ceb9ba193" candidate_uuid: "6acde597072043feb32cb932fd0d4a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8fd7489dc34911b207d2513067b009" is_pre_election: true
I20260809 09:31:33.076646 10574 raft_consensus.cc:2454] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6acde597072043feb32cb932fd0d4a72 in term 0.
I20260809 09:31:33.076664 10128 raft_consensus.cc:2454] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6acde597072043feb32cb932fd0d4a72 in term 0.
I20260809 09:31:33.077101 10236 leader_election.cc:302] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6acde597072043feb32cb932fd0d4a72, c8a74cf1b6bc48fa85999359b0311afc; no voters:
I20260809 09:31:33.077361 10700 raft_consensus.cc:2781] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:31:33.077472 10700 raft_consensus.cc:482] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:31:33.077528 10700 raft_consensus.cc:3037] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:33.078480 10700 raft_consensus.cc:504] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6acde597072043feb32cb932fd0d4a72" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 43019 } } peers { permanent_uuid: "ba8fd7489dc34911b207d2513067b009" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 40067 } } peers { permanent_uuid: "c8a74cf1b6bc48fa85999359b0311afc" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38645 } }
I20260809 09:31:33.078840 10700 leader_election.cc:288] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [CANDIDATE]: Term 1 election: Requested vote from peers ba8fd7489dc34911b207d2513067b009 (127.9.102.1:40067), c8a74cf1b6bc48fa85999359b0311afc (127.9.102.3:38645)
I20260809 09:31:33.079159 10128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc06a07548a43618c16262ceb9ba193" candidate_uuid: "6acde597072043feb32cb932fd0d4a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8fd7489dc34911b207d2513067b009"
I20260809 09:31:33.079216 10574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc06a07548a43618c16262ceb9ba193" candidate_uuid: "6acde597072043feb32cb932fd0d4a72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a74cf1b6bc48fa85999359b0311afc"
I20260809 09:31:33.079269 10128 raft_consensus.cc:3037] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:33.079331 10574 raft_consensus.cc:3037] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:33.080549 10128 raft_consensus.cc:2454] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6acde597072043feb32cb932fd0d4a72 in term 1.
I20260809 09:31:33.080557 10574 raft_consensus.cc:2454] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6acde597072043feb32cb932fd0d4a72 in term 1.
I20260809 09:31:33.080848 10239 leader_election.cc:302] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6acde597072043feb32cb932fd0d4a72, ba8fd7489dc34911b207d2513067b009; no voters:
I20260809 09:31:33.081012 10700 raft_consensus.cc:2781] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:33.081194 10700 raft_consensus.cc:686] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 1 LEADER]: Becoming Leader. State: Replica: 6acde597072043feb32cb932fd0d4a72, State: Running, Role: LEADER
I20260809 09:31:33.081498 10700 consensus_queue.cc:237] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6acde597072043feb32cb932fd0d4a72" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 43019 } } peers { permanent_uuid: "ba8fd7489dc34911b207d2513067b009" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 40067 } } peers { permanent_uuid: "c8a74cf1b6bc48fa85999359b0311afc" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38645 } }
I20260809 09:31:33.084453 9893 catalog_manager.cc:5495] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6acde597072043feb32cb932fd0d4a72 (127.9.102.2). New cstate: current_term: 1 leader_uuid: "6acde597072043feb32cb932fd0d4a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6acde597072043feb32cb932fd0d4a72" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 43019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba8fd7489dc34911b207d2513067b009" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 40067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a74cf1b6bc48fa85999359b0311afc" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38645 } health_report { overall_health: UNKNOWN } } }
I20260809 09:31:33.152215 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.149335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.9.102.2:43019)
0809 09:31:33.149443 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:33.149446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:33.149458 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:31:33.149536 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:33.149538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:33.149581 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:33.149658 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:31:33.149868 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.150348 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.151093 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.151270 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.151337 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.151568 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:31:33.151893 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:31:33.151974 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:33.152027 (+ 53us) server_negotiation.cc:299] Negotiation successful
0809 09:31:33.152074 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:31:33.157255 10574 raft_consensus.cc:1261] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc [term 1 FOLLOWER]: Refusing update from remote peer 6acde597072043feb32cb932fd0d4a72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:31:33.157260 10128 raft_consensus.cc:1261] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009 [term 1 FOLLOWER]: Refusing update from remote peer 6acde597072043feb32cb932fd0d4a72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:31:33.157917 10700 consensus_queue.cc:1035] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8fd7489dc34911b207d2513067b009" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 40067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:31:33.158092 10703 consensus_queue.cc:1035] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a74cf1b6bc48fa85999359b0311afc" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 38645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:31:33.165445 10711 mvcc.cc:205] Tried to move back new op lower bound from 7316553290365976576 to 7316553290064703488. Current Snapshot: MvccSnapshot[applied={T|T < 7316553290365976576}]
I20260809 09:31:33.165588 10713 mvcc.cc:205] Tried to move back new op lower bound from 7316553290365976576 to 7316553290064703488. Current Snapshot: MvccSnapshot[applied={T|T < 7316553290365976576}]
I20260809 09:31:33.167284 10715 mvcc.cc:205] Tried to move back new op lower bound from 7316553290365976576 to 7316553290064703488. Current Snapshot: MvccSnapshot[applied={T|T < 7316553290365976576}]
I20260809 09:31:33.172608 9893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:31:33.172763 9893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:31:33.175061 9893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3cbf47f563e4e178168d6551c657543: Sending DeleteTablet for 3 replicas of tablet ffc06a07548a43618c16262ceb9ba193
I20260809 09:31:33.175715 10307 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc06a07548a43618c16262ceb9ba193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:33 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:57848
I20260809 09:31:33.175730 10092 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc06a07548a43618c16262ceb9ba193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:33 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:49592
I20260809 09:31:33.176162 10728 tablet_replica.cc:330] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009: stopping tablet replica
I20260809 09:31:33.176249 10731 tablet_replica.cc:330] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72: stopping tablet replica
I20260809 09:31:33.176437 10541 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc06a07548a43618c16262ceb9ba193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:33 UTC) from {username='kudu', principal='kudu/127.9.102.62@KRBTEST.COM'} at 127.0.0.1:47096
I20260809 09:31:33.176489 10731 raft_consensus.cc:2229] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 1 LEADER]: Raft consensus shutting down.
I20260809 09:31:33.176489 10728 raft_consensus.cc:2229] T ffc06a07548a43618c16262ceb9ba193 P ba8fd7489dc34911b207d2513067b009 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:31:33.176750 10731 raft_consensus.cc:2258] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:33.177026 10735 tablet_replica.cc:330] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc: stopping tablet replica
I20260809 09:31:33.177279 10735 raft_consensus.cc:2229] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:31:33.177536 10735 raft_consensus.cc:2258] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:33.177639 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 9997
I20260809 09:31:33.181103 10731 ts_tablet_manager.cc:1905] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:31:33.181666 10735 ts_tablet_manager.cc:1905] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:31:33.184134 10735 ts_tablet_manager.cc:1918] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:31:33.184221 10735 log.cc:1196] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/ffc06a07548a43618c16262ceb9ba193
I20260809 09:31:33.184548 10735 ts_tablet_manager.cc:1939] T ffc06a07548a43618c16262ceb9ba193 P c8a74cf1b6bc48fa85999359b0311afc: Deleting consensus metadata
I20260809 09:31:33.184541 10731 ts_tablet_manager.cc:1918] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:31:33.184646 10731 log.cc:1196] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/ffc06a07548a43618c16262ceb9ba193
I20260809 09:31:33.184995 10731 ts_tablet_manager.cc:1939] T ffc06a07548a43618c16262ceb9ba193 P 6acde597072043feb32cb932fd0d4a72: Deleting consensus metadata
I20260809 09:31:33.185580 9861 catalog_manager.cc:4841] TS c8a74cf1b6bc48fa85999359b0311afc (127.9.102.3:38645): tablet ffc06a07548a43618c16262ceb9ba193 (table test-table [id=6a14c485f73b46ccb7101eb59eabeb9e]) successfully deleted
I20260809 09:31:33.185933 9865 catalog_manager.cc:4841] TS 6acde597072043feb32cb932fd0d4a72 (127.9.102.2:43019): tablet ffc06a07548a43618c16262ceb9ba193 (table test-table [id=6a14c485f73b46ccb7101eb59eabeb9e]) successfully deleted
W20260809 09:31:33.190953 9866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:40067 (error 108)
I20260809 09:31:33.191138 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 10202
I20260809 09:31:33.196899 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.196672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:40067 (local address 127.0.0.1:49596)
0809 09:31:33.196782 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:33.196837 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:40067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 09:31:33.197130 9866 catalog_manager.cc:4584] TS ba8fd7489dc34911b207d2513067b009 (127.9.102.1:40067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffc06a07548a43618c16262ceb9ba193: Network error: Client connection negotiation failed: client connection to 127.9.102.1:40067: connect: Connection refused (error 111)
I20260809 09:31:33.200470 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 10436
I20260809 09:31:33.209239 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 9830
2026-08-09T09:31:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2532 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T09:31:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:33Z Disabled control of system clock
I20260809 09:31:33.244503 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:45453
--webserver_interface=127.9.102.62
--webserver_port=0
--builtin_ntp_servers=127.9.102.20:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:45453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 09:31:33.342875 10748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:33.343114 10748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:33.343186 10748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:33.346834 10748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:33.346908 10748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:33.346937 10748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:33.346971 10748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:33.347002 10748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:33.351122 10748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:45453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:45453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:33.351928 10748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:33.353072 10748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:33.359958 10753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:33.359957 10757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:33.360014 10755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:33.360082 10748 server_base.cc:1034] running on GCE node
I20260809 09:31:33.360683 10748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:33.361615 10748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:33.362799 10748 hybrid_clock.cc:648] HybridClock initialized: now 1786267893362772 us; error 39 us; skew 500 ppm
I20260809 09:31:33.364423 10748 webserver.cc:460] Webserver started at http://127.9.102.62:44897/ using document root <none> and password file <none>
I20260809 09:31:33.364912 10748 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:33.364998 10748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:33.365209 10748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:33.366894 10748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "fdc0f8f7a85b4005b19fad9522c3fb47"
format_stamp: "Formatted at 2026-08-09 09:31:33 on dist-test-slave-7vjv"
server_key: "6e90f53f8f7de75999487ad52abc4f24"
server_key_iv: "a63b66cf5fbe64e9eadaa7ebf5546fd6"
server_key_version: "encryptionkey@0"
I20260809 09:31:33.367393 10748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "fdc0f8f7a85b4005b19fad9522c3fb47"
format_stamp: "Formatted at 2026-08-09 09:31:33 on dist-test-slave-7vjv"
server_key: "6e90f53f8f7de75999487ad52abc4f24"
server_key_iv: "a63b66cf5fbe64e9eadaa7ebf5546fd6"
server_key_version: "encryptionkey@0"
I20260809 09:31:33.370653 10748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260809 09:31:33.372721 10771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:33.373863 10748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 09:31:33.374001 10748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "fdc0f8f7a85b4005b19fad9522c3fb47"
format_stamp: "Formatted at 2026-08-09 09:31:33 on dist-test-slave-7vjv"
server_key: "6e90f53f8f7de75999487ad52abc4f24"
server_key_iv: "a63b66cf5fbe64e9eadaa7ebf5546fd6"
server_key_version: "encryptionkey@0"
I20260809 09:31:33.374102 10748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:33.383527 10748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:33.384037 10748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:33.384282 10748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:33.390693 10748 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:45453
I20260809 09:31:33.390704 10858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:45453 every 8 connection(s)
I20260809 09:31:33.391472 10748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:33.394177 10859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:33.398936 10859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47: Bootstrap starting.
I20260809 09:31:33.399811 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 10748
I20260809 09:31:33.399948 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:33.400291 9624 external_mini_cluster.cc:1428] Setting key 44badf15a557cd73b36250ff0096650e
I20260809 09:31:33.401643 10859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:33.402316 10859 log.cc:824] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:33.403952 10859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47: No bootstrap required, opened a new log
I20260809 09:31:33.407088 10859 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc0f8f7a85b4005b19fad9522c3fb47" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45453 } }
I20260809 09:31:33.407303 10859 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:33.407384 10859 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc0f8f7a85b4005b19fad9522c3fb47, State: Initialized, Role: FOLLOWER
I20260809 09:31:33.407843 10859 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc0f8f7a85b4005b19fad9522c3fb47" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45453 } }
I20260809 09:31:33.407958 10859 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:33.408061 10859 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:33.408213 10859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:33.408600 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.401251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.9.102.62:45453)
0809 09:31:33.401835 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:33.401850 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:33.401880 (+ 30us) server_negotiation.cc:407] Connection header received
0809 09:31:33.402416 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:33.402436 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:33.402849 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:33.403191 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:33.403566 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.404507 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.405315 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.405746 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.405902 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.406777 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:33.406782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:33.406793 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:33.407126 (+ 333us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:33.407173 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:33.407192 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:33.407299 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:33.407511 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 09:31:33.407782 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260809 09:31:33.409081 10859 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc0f8f7a85b4005b19fad9522c3fb47" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45453 } }
I20260809 09:31:33.409401 10859 leader_election.cc:302] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc0f8f7a85b4005b19fad9522c3fb47; no voters:
I20260809 09:31:33.409699 10859 leader_election.cc:288] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:33.409808 10871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:33.410012 10871 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [term 1 LEADER]: Becoming Leader. State: Replica: fdc0f8f7a85b4005b19fad9522c3fb47, State: Running, Role: LEADER
I20260809 09:31:33.410275 10871 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc0f8f7a85b4005b19fad9522c3fb47" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45453 } }
I20260809 09:31:33.410777 10859 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:33.411334 10874 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdc0f8f7a85b4005b19fad9522c3fb47. Latest consensus state: current_term: 1 leader_uuid: "fdc0f8f7a85b4005b19fad9522c3fb47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc0f8f7a85b4005b19fad9522c3fb47" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45453 } } }
I20260809 09:31:33.411468 10874 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:33.411734 10875 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdc0f8f7a85b4005b19fad9522c3fb47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc0f8f7a85b4005b19fad9522c3fb47" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45453 } } }
I20260809 09:31:33.411836 10875 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:33.412215 10883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:33.415287 10883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:33.420679 10883 catalog_manager.cc:1347] Generated new cluster ID: 15f10ff3e224454b9057c723b1dc65c7
I20260809 09:31:33.420753 10883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:33.435344 10883 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:33.436265 10883 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:33.442858 10883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47: Generated new TSK 0
I20260809 09:31:33.443404 10883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:33.448956 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.1:0
--local_ip_for_outbound_sockets=127.9.102.1
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:45453
--builtin_ntp_servers=127.9.102.20:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 09:31:33.562958 10904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:33.563155 10904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:33.563186 10904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:33.566783 10904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:33.566824 10904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:33.566906 10904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.1
I20260809 09:31:33.571055 10904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.102.1
--webserver_port=0
--tserver_master_addrs=127.9.102.62:45453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.1
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:33.571856 10904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:33.573055 10904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:33.582059 10919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:33.582095 10914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:33.582047 10912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:33.582356 10904 server_base.cc:1034] running on GCE node
I20260809 09:31:33.582716 10904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:33.583256 10904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:33.584403 10904 hybrid_clock.cc:648] HybridClock initialized: now 1786267893584387 us; error 27 us; skew 500 ppm
I20260809 09:31:33.589689 10904 webserver.cc:460] Webserver started at http://127.9.102.1:42025/ using document root <none> and password file <none>
I20260809 09:31:33.590174 10904 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:33.590227 10904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:33.590443 10904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:33.592334 10904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/data/instance:
uuid: "ab5acafba2a44d58ba792152e6c512b5"
format_stamp: "Formatted at 2026-08-09 09:31:33 on dist-test-slave-7vjv"
server_key: "d6ec44a01a3391fc413a5ee0a9a175f5"
server_key_iv: "63f1cffbceb95e2c248b3204791b6fd5"
server_key_version: "encryptionkey@0"
I20260809 09:31:33.592860 10904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance:
uuid: "ab5acafba2a44d58ba792152e6c512b5"
format_stamp: "Formatted at 2026-08-09 09:31:33 on dist-test-slave-7vjv"
server_key: "d6ec44a01a3391fc413a5ee0a9a175f5"
server_key_iv: "63f1cffbceb95e2c248b3204791b6fd5"
server_key_version: "encryptionkey@0"
I20260809 09:31:33.596064 10904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:31:33.598322 10929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:33.599326 10904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:33.599442 10904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/wal
uuid: "ab5acafba2a44d58ba792152e6c512b5"
format_stamp: "Formatted at 2026-08-09 09:31:33 on dist-test-slave-7vjv"
server_key: "d6ec44a01a3391fc413a5ee0a9a175f5"
server_key_iv: "63f1cffbceb95e2c248b3204791b6fd5"
server_key_version: "encryptionkey@0"
I20260809 09:31:33.599532 10904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:33.610913 10904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:33.611481 10904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:33.611661 10904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:33.612278 10904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:33.613094 10904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:33.613163 10904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:33.613224 10904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:33.613270 10904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:33.621430 10904 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.1:38317
I20260809 09:31:33.621456 11120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.1:38317 every 8 connection(s)
I20260809 09:31:33.622175 10904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/data/info.pb
I20260809 09:31:33.626049 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 10904
I20260809 09:31:33.626195 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-0/wal/instance
I20260809 09:31:33.626466 9624 external_mini_cluster.cc:1428] Setting key fcc66e8a3019bbd66b1074ca838b5fdf
I20260809 09:31:33.628371 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.2:0
--local_ip_for_outbound_sockets=127.9.102.2
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:45453
--builtin_ntp_servers=127.9.102.20:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 09:31:33.630772 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.623740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:60357 (local address 127.9.102.62:45453)
0809 09:31:33.623878 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:33.623882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:33.624793 (+ 911us) server_negotiation.cc:407] Connection header received
0809 09:31:33.626046 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:33.626050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:33.626112 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:33.626219 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:33.627258 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.627876 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.629070 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.629279 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.629385 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.630319 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:33.630322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:33.630325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:33.630342 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:33.630359 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:33.630367 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:33.630439 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:33.630590 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 09:31:33.630647 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 09:31:33.631510 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.624053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:45453 (local address 127.9.102.1:60357)
0809 09:31:33.624659 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:33.624694 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:33.625836 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:33.626420 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:33.626427 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:33.626528 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:33.627080 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.627096 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.628004 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.628008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:33.628941 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.628951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.629418 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.629423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:33.629570 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.630082 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:31:33.630083 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:33.630117 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:31:33.630170 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:33.630441 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:33.630453 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:33.630461 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 09:31:33.630587 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 09:31:33.630813 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260809 09:31:33.632664 11121 heartbeater.cc:347] Connected to a master server at 127.9.102.62:45453
I20260809 09:31:33.632884 11121 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:33.633452 11121 heartbeater.cc:511] Master 127.9.102.62:45453 requested a full tablet report, sending...
I20260809 09:31:33.635151 10802 ts_manager.cc:194] Registered new tserver with Master: ab5acafba2a44d58ba792152e6c512b5 (127.9.102.1:38317)
I20260809 09:31:33.636722 10802 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.102.1:60357
W20260809 09:31:33.733021 11124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:33.733219 11124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:33.733249 11124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:33.736451 11124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:33.736485 11124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:33.736533 11124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.2
I20260809 09:31:33.740267 11124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.102.2
--webserver_port=0
--tserver_master_addrs=127.9.102.62:45453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.2
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:33.740950 11124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:33.741950 11124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:33.750080 11136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:33.750168 11138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:33.750087 11135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:33.750161 11124 server_base.cc:1034] running on GCE node
I20260809 09:31:33.750587 11124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:33.751092 11124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:33.752226 11124 hybrid_clock.cc:648] HybridClock initialized: now 1786267893752211 us; error 26 us; skew 500 ppm
I20260809 09:31:33.753742 11124 webserver.cc:460] Webserver started at http://127.9.102.2:45073/ using document root <none> and password file <none>
I20260809 09:31:33.754207 11124 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:33.754262 11124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:33.754478 11124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:33.756210 11124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/data/instance:
uuid: "28dfb88728f846bc9021c397222b08df"
format_stamp: "Formatted at 2026-08-09 09:31:33 on dist-test-slave-7vjv"
server_key: "59635736df4a318555fb58e431a999a8"
server_key_iv: "7b125b8257d32744b144f95ed1a3cde1"
server_key_version: "encryptionkey@0"
I20260809 09:31:33.756693 11124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance:
uuid: "28dfb88728f846bc9021c397222b08df"
format_stamp: "Formatted at 2026-08-09 09:31:33 on dist-test-slave-7vjv"
server_key: "59635736df4a318555fb58e431a999a8"
server_key_iv: "7b125b8257d32744b144f95ed1a3cde1"
server_key_version: "encryptionkey@0"
I20260809 09:31:33.759642 11124 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:31:33.761659 11149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:33.762589 11124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:31:33.762708 11124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/wal
uuid: "28dfb88728f846bc9021c397222b08df"
format_stamp: "Formatted at 2026-08-09 09:31:33 on dist-test-slave-7vjv"
server_key: "59635736df4a318555fb58e431a999a8"
server_key_iv: "7b125b8257d32744b144f95ed1a3cde1"
server_key_version: "encryptionkey@0"
I20260809 09:31:33.762801 11124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:33.781090 11124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:33.781606 11124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:33.781764 11124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:33.782231 11124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:33.783010 11124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:33.783077 11124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:33.783140 11124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:33.783186 11124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:33.790853 11124 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.2:44787
I20260809 09:31:33.790871 11347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.2:44787 every 8 connection(s)
I20260809 09:31:33.791532 11124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/data/info.pb
I20260809 09:31:33.795745 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 11124
I20260809 09:31:33.795861 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/wal/instance
I20260809 09:31:33.796223 9624 external_mini_cluster.cc:1428] Setting key 73497d1cf5601baf7fd172ce1b83b382
I20260809 09:31:33.798046 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.102.3:0
--local_ip_for_outbound_sockets=127.9.102.3
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:45453
--builtin_ntp_servers=127.9.102.20:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 09:31:33.799568 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.793046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.2:36411 (local address 127.9.102.62:45453)
0809 09:31:33.793166 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:33.793170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:33.793920 (+ 750us) server_negotiation.cc:407] Connection header received
0809 09:31:33.795026 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:33.795028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:33.795070 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:33.795133 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:33.795990 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.796630 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.797845 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.798033 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.798103 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.799172 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:33.799175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:33.799176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:33.799187 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:33.799203 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:33.799210 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:33.799265 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:33.799425 (+ 160us) server_negotiation.cc:299] Negotiation successful
0809 09:31:33.799468 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260809 09:31:33.800458 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.793275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:45453 (local address 127.9.102.2:36411)
0809 09:31:33.793817 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:33.793847 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:33.794871 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:33.795255 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:33.795260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:33.795327 (+ 67us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:33.795861 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.795881 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.796783 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.796786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:33.797737 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.797747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.798178 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.798184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:33.798376 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.798895 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:31:33.798899 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:33.798942 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:31:33.799030 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:33.799289 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:33.799300 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:33.799308 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 09:31:33.799420 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 09:31:33.799686 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
I20260809 09:31:33.801409 11349 heartbeater.cc:347] Connected to a master server at 127.9.102.62:45453
I20260809 09:31:33.801610 11349 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:33.802094 11349 heartbeater.cc:511] Master 127.9.102.62:45453 requested a full tablet report, sending...
I20260809 09:31:33.803143 10802 ts_manager.cc:194] Registered new tserver with Master: 28dfb88728f846bc9021c397222b08df (127.9.102.2:44787)
I20260809 09:31:33.803655 10802 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.102.2:36411
W20260809 09:31:33.897471 11353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:33.897661 11353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:33.897691 11353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:33.901394 11353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:33.901437 11353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:33.901484 11353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.102.3
I20260809 09:31:33.905642 11353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.102.3
--webserver_port=0
--tserver_master_addrs=127.9.102.62:45453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.102.3
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:33.906410 11353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:33.907493 11353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:33.916113 11361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:33.916142 11364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:33.916260 11353 server_base.cc:1034] running on GCE node
W20260809 09:31:33.916152 11362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:33.916623 11353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:33.917096 11353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:33.918243 11353 hybrid_clock.cc:648] HybridClock initialized: now 1786267893918257 us; error 51 us; skew 500 ppm
I20260809 09:31:33.919870 11353 webserver.cc:460] Webserver started at http://127.9.102.3:44991/ using document root <none> and password file <none>
I20260809 09:31:33.920398 11353 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:33.920457 11353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:33.920657 11353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:33.922288 11353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/data/instance:
uuid: "f7a44a5f516545269a3562aad6204c62"
format_stamp: "Formatted at 2026-08-09 09:31:33 on dist-test-slave-7vjv"
server_key: "ec30f5b4a8e91eba2c37bdab2b19bbc8"
server_key_iv: "514d4fe6f3638a6c4e6c1458a3fb4d68"
server_key_version: "encryptionkey@0"
I20260809 09:31:33.922767 11353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance:
uuid: "f7a44a5f516545269a3562aad6204c62"
format_stamp: "Formatted at 2026-08-09 09:31:33 on dist-test-slave-7vjv"
server_key: "ec30f5b4a8e91eba2c37bdab2b19bbc8"
server_key_iv: "514d4fe6f3638a6c4e6c1458a3fb4d68"
server_key_version: "encryptionkey@0"
I20260809 09:31:33.925833 11353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:31:33.927924 11377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:33.928958 11353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:33.929081 11353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/wal
uuid: "f7a44a5f516545269a3562aad6204c62"
format_stamp: "Formatted at 2026-08-09 09:31:33 on dist-test-slave-7vjv"
server_key: "ec30f5b4a8e91eba2c37bdab2b19bbc8"
server_key_iv: "514d4fe6f3638a6c4e6c1458a3fb4d68"
server_key_version: "encryptionkey@0"
I20260809 09:31:33.929179 11353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:33.948613 11353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:33.949112 11353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:33.949285 11353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:33.949733 11353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 09:31:33.950496 11353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 09:31:33.950567 11353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:33.950629 11353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 09:31:33.950672 11353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:33.958429 11353 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.3:33581
I20260809 09:31:33.958448 11576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.3:33581 every 8 connection(s)
I20260809 09:31:33.959195 11353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/data/info.pb
I20260809 09:31:33.965338 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 11353
I20260809 09:31:33.965437 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/wal/instance
I20260809 09:31:33.965660 9624 external_mini_cluster.cc:1428] Setting key c61adf9e82c33490061d9781013391e2
I20260809 09:31:33.967052 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.960803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.3:34587 (local address 127.9.102.62:45453)
0809 09:31:33.960919 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:33.960922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:33.961738 (+ 816us) server_negotiation.cc:407] Connection header received
0809 09:31:33.962895 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:33.962898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:33.962952 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:33.963019 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:33.963735 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.964263 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.965224 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.965474 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.965564 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.966547 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:33.966550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:33.966552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:33.966563 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:33.966583 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:33.966591 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:33.966685 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:33.966835 (+ 150us) server_negotiation.cc:299] Negotiation successful
0809 09:31:33.966890 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 09:31:33.967790 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.961099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:45453 (local address 127.9.102.3:34587)
0809 09:31:33.961615 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:33.961647 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:33.962709 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:33.963158 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:33.963164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:33.963232 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:31:33.963606 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.963617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.964378 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.964381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:33.965107 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.965115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.965596 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.965600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:33.965770 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.966324 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 09:31:33.966325 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:31:33.966357 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 09:31:33.966408 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:31:33.966698 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:31:33.966707 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:31:33.966712 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 09:31:33.966834 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 09:31:33.967064 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260809 09:31:33.968735 11577 heartbeater.cc:347] Connected to a master server at 127.9.102.62:45453
I20260809 09:31:33.968915 11577 heartbeater.cc:464] Registering TS with master...
I20260809 09:31:33.969333 11577 heartbeater.cc:511] Master 127.9.102.62:45453 requested a full tablet report, sending...
I20260809 09:31:33.970177 10802 ts_manager.cc:194] Registered new tserver with Master: f7a44a5f516545269a3562aad6204c62 (127.9.102.3:33581)
I20260809 09:31:33.970671 10802 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.102.3:34587
I20260809 09:31:33.979012 9624 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 09:31:33.983767 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.980852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.9.102.62:45453)
0809 09:31:33.981018 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:33.981034 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:33.981120 (+ 86us) server_negotiation.cc:407] Connection header received
0809 09:31:33.981198 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:33.981199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:33.981232 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:33.981289 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:33.981488 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.981880 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.982549 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.982716 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.982780 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:33.983372 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:33.983375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:33.983377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:33.983386 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:33.983401 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:33.983408 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:33.983460 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:33.983561 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 09:31:33.983603 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 09:31:33.986065 10802 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 09:31:33.988855 10802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 09:31:34.000944 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.995965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:44787 (local address 127.0.0.1:39310)
0809 09:31:33.996420 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:33.996431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:33.996494 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:33.996983 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:33.996988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:33.997012 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:33.997114 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.997120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.998393 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.998396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:33.999500 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.999510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.000483 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.000486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:34.000580 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:34.000617 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 09:31:34.000714 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 09:31:34.000820 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":107,"threads_started":1}
I20260809 09:31:34.001022 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.996266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.9.102.2:44787)
0809 09:31:33.996556 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:33.996566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:33.996589 (+ 23us) server_negotiation.cc:407] Connection header received
0809 09:31:33.996659 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:33.996663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:33.996847 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:33.996998 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:33.997325 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.998285 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.999627 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.000366 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.000469 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:34.000650 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:34.000763 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 09:31:34.000874 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":126,"threads_started":1}
I20260809 09:31:34.001190 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.995851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:33581 (local address 127.0.0.1:40316)
0809 09:31:33.996280 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:33.996317 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:33.996419 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:33.998082 (+ 1663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:33.998085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:33.998101 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:33.998177 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.998181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.999090 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.999096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:33.999852 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.999862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.000825 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.000828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:34.000916 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:34.000941 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 09:31:34.000987 (+ 46us) client_negotiation.cc:240] Negotiation successful
0809 09:31:34.001066 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":72,"threads_started":1}
I20260809 09:31:34.001750 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.995923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.9.102.3:33581)
0809 09:31:33.997716 (+ 1793us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:33.997726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:33.997744 (+ 18us) server_negotiation.cc:407] Connection header received
0809 09:31:33.997796 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:33.997800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:33.997945 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:33.998289 (+ 344us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:33.998412 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.998981 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.999981 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.000691 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.001346 (+ 655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:34.001513 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:34.001555 (+ 42us) server_negotiation.cc:299] Negotiation successful
0809 09:31:34.001625 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1714,"thread_start_us":106,"threads_started":1}
I20260809 09:31:34.001886 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.996937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38317 (local address 127.0.0.1:39254)
0809 09:31:33.997424 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:33.997436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:33.997511 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:33.997984 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:33.997987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:33.998003 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:33.998074 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:33.998080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.999224 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:33.999227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:34.000614 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:34.000621 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.001563 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.001566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:34.001638 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:34.001656 (+ 18us) client_negotiation.cc:769] Sending connection context
0809 09:31:34.001707 (+ 51us) client_negotiation.cc:240] Negotiation successful
0809 09:31:34.001767 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":82,"threads_started":1}
I20260809 09:31:34.002018 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:33.997333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.9.102.1:38317)
0809 09:31:33.997638 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:33.997647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:33.997672 (+ 25us) server_negotiation.cc:407] Connection header received
0809 09:31:33.997727 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:33.997730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:33.997847 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:33.997994 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:33.998158 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:33.999108 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.000756 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.001478 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.001564 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:34.001719 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:34.001758 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 09:31:34.001831 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":150,"threads_started":1}
I20260809 09:31:34.004137 11223 tablet_service.cc:1467] Processing CreateTablet for tablet 56a27a00a0ac45d196a9cf96d5669521 (DEFAULT_TABLE table=test-table [id=b8b59664e7f046b894bc0cd929f020ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:34.004530 11470 tablet_service.cc:1467] Processing CreateTablet for tablet 56a27a00a0ac45d196a9cf96d5669521 (DEFAULT_TABLE table=test-table [id=b8b59664e7f046b894bc0cd929f020ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:34.005216 11223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a27a00a0ac45d196a9cf96d5669521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:34.005442 11470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a27a00a0ac45d196a9cf96d5669521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:34.005752 11007 tablet_service.cc:1467] Processing CreateTablet for tablet 56a27a00a0ac45d196a9cf96d5669521 (DEFAULT_TABLE table=test-table [id=b8b59664e7f046b894bc0cd929f020ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 09:31:34.006538 11007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a27a00a0ac45d196a9cf96d5669521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:34.011555 11609 tablet_bootstrap.cc:492] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5: Bootstrap starting.
I20260809 09:31:34.012777 11614 tablet_bootstrap.cc:492] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df: Bootstrap starting.
I20260809 09:31:34.012776 11619 tablet_bootstrap.cc:492] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62: Bootstrap starting.
I20260809 09:31:34.013789 11609 tablet_bootstrap.cc:654] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:34.014393 11614 tablet_bootstrap.cc:654] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:34.014566 11609 log.cc:824] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:34.014622 11619 tablet_bootstrap.cc:654] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:34.015125 11614 log.cc:824] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:34.015288 11619 log.cc:824] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:34.016749 11614 tablet_bootstrap.cc:492] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df: No bootstrap required, opened a new log
I20260809 09:31:34.016953 11614 ts_tablet_manager.cc:1397] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 09:31:34.016975 11609 tablet_bootstrap.cc:492] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5: No bootstrap required, opened a new log
I20260809 09:31:34.017164 11619 tablet_bootstrap.cc:492] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62: No bootstrap required, opened a new log
I20260809 09:31:34.017168 11609 ts_tablet_manager.cc:1397] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 09:31:34.017314 11619 ts_tablet_manager.cc:1397] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 09:31:34.019615 11614 raft_consensus.cc:348] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a44a5f516545269a3562aad6204c62" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33581 } } peers { permanent_uuid: "28dfb88728f846bc9021c397222b08df" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 44787 } } peers { permanent_uuid: "ab5acafba2a44d58ba792152e6c512b5" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38317 } }
I20260809 09:31:34.019814 11614 raft_consensus.cc:374] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:34.019901 11614 raft_consensus.cc:729] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28dfb88728f846bc9021c397222b08df, State: Initialized, Role: FOLLOWER
I20260809 09:31:34.020339 11614 consensus_queue.cc:260] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a44a5f516545269a3562aad6204c62" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33581 } } peers { permanent_uuid: "28dfb88728f846bc9021c397222b08df" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 44787 } } peers { permanent_uuid: "ab5acafba2a44d58ba792152e6c512b5" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38317 } }
I20260809 09:31:34.020565 11609 raft_consensus.cc:348] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a44a5f516545269a3562aad6204c62" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33581 } } peers { permanent_uuid: "28dfb88728f846bc9021c397222b08df" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 44787 } } peers { permanent_uuid: "ab5acafba2a44d58ba792152e6c512b5" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38317 } }
I20260809 09:31:34.020838 11609 raft_consensus.cc:374] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:34.020933 11609 raft_consensus.cc:729] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5acafba2a44d58ba792152e6c512b5, State: Initialized, Role: FOLLOWER
I20260809 09:31:34.020984 11349 heartbeater.cc:503] Master 127.9.102.62:45453 was elected leader, sending a full tablet report...
I20260809 09:31:34.020861 11619 raft_consensus.cc:348] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a44a5f516545269a3562aad6204c62" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33581 } } peers { permanent_uuid: "28dfb88728f846bc9021c397222b08df" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 44787 } } peers { permanent_uuid: "ab5acafba2a44d58ba792152e6c512b5" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38317 } }
I20260809 09:31:34.021101 11619 raft_consensus.cc:374] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:34.021148 11619 raft_consensus.cc:729] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a44a5f516545269a3562aad6204c62, State: Initialized, Role: FOLLOWER
I20260809 09:31:34.021341 11609 consensus_queue.cc:260] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a44a5f516545269a3562aad6204c62" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33581 } } peers { permanent_uuid: "28dfb88728f846bc9021c397222b08df" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 44787 } } peers { permanent_uuid: "ab5acafba2a44d58ba792152e6c512b5" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38317 } }
I20260809 09:31:34.021430 11619 consensus_queue.cc:260] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a44a5f516545269a3562aad6204c62" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33581 } } peers { permanent_uuid: "28dfb88728f846bc9021c397222b08df" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 44787 } } peers { permanent_uuid: "ab5acafba2a44d58ba792152e6c512b5" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38317 } }
I20260809 09:31:34.021919 11614 ts_tablet_manager.cc:1428] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260809 09:31:34.022147 11619 ts_tablet_manager.cc:1428] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260809 09:31:34.022147 11609 ts_tablet_manager.cc:1428] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260809 09:31:34.022501 11121 heartbeater.cc:503] Master 127.9.102.62:45453 was elected leader, sending a full tablet report...
I20260809 09:31:34.022789 11577 heartbeater.cc:503] Master 127.9.102.62:45453 was elected leader, sending a full tablet report...
W20260809 09:31:34.042436 11350 tablet.cc:2369] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:31:34.050695 11628 raft_consensus.cc:482] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 09:31:34.050858 11628 raft_consensus.cc:504] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a44a5f516545269a3562aad6204c62" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33581 } } peers { permanent_uuid: "28dfb88728f846bc9021c397222b08df" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 44787 } } peers { permanent_uuid: "ab5acafba2a44d58ba792152e6c512b5" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38317 } }
I20260809 09:31:34.051681 11628 leader_election.cc:288] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7a44a5f516545269a3562aad6204c62 (127.9.102.3:33581), 28dfb88728f846bc9021c397222b08df (127.9.102.2:44787)
I20260809 09:31:34.054960 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:34.051865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.2:44787 (local address 127.9.102.1:38953)
0809 09:31:34.052041 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:34.052055 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:34.052189 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:34.052442 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:34.052445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:34.052460 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:34.052579 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:34.052586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.053282 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.053287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:34.053983 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:34.053992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.054573 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.054577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:34.054662 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:34.054687 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 09:31:34.054747 (+ 60us) client_negotiation.cc:240] Negotiation successful
0809 09:31:34.054822 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":24}
I20260809 09:31:34.054963 11601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:34.051924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:38953 (local address 127.9.102.2:44787)
0809 09:31:34.052019 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:34.052023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:34.052123 (+ 100us) server_negotiation.cc:407] Connection header received
0809 09:31:34.052288 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:34.052293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:34.052343 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:34.052412 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:34.052688 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.053178 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.054102 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.054457 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.054531 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:34.054604 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:34.054789 (+ 185us) server_negotiation.cc:299] Negotiation successful
0809 09:31:34.054860 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 09:31:34.054991 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:34.051865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.3:33581 (local address 127.9.102.1:59903)
0809 09:31:34.052247 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:34.052259 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:31:34.052353 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:31:34.052586 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:31:34.052588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:31:34.052601 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 09:31:34.052730 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:34.052737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.053350 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.053352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:34.054018 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:31:34.054027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.054630 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.054634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:31:34.054707 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:34.054722 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 09:31:34.054809 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 09:31:34.054860 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":41,"thread_start_us":92,"threads_started":1}
I20260809 09:31:34.054993 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:34.051924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.102.1:59903 (local address 127.9.102.3:33581)
0809 09:31:34.052019 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:34.052037 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:34.052337 (+ 300us) server_negotiation.cc:407] Connection header received
0809 09:31:34.052449 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:34.052452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:34.052490 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:34.052567 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 09:31:34.052833 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.053241 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.054158 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.054479 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.054603 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:34.054655 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:34.054840 (+ 185us) server_negotiation.cc:299] Negotiation successful
0809 09:31:34.054910 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260809 09:31:34.055521 11506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a27a00a0ac45d196a9cf96d5669521" candidate_uuid: "ab5acafba2a44d58ba792152e6c512b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a44a5f516545269a3562aad6204c62" is_pre_election: true
I20260809 09:31:34.055519 11255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a27a00a0ac45d196a9cf96d5669521" candidate_uuid: "ab5acafba2a44d58ba792152e6c512b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dfb88728f846bc9021c397222b08df" is_pre_election: true
I20260809 09:31:34.055751 11506 raft_consensus.cc:2454] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5acafba2a44d58ba792152e6c512b5 in term 0.
I20260809 09:31:34.055751 11255 raft_consensus.cc:2454] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5acafba2a44d58ba792152e6c512b5 in term 0.
I20260809 09:31:34.056226 10935 leader_election.cc:302] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28dfb88728f846bc9021c397222b08df, ab5acafba2a44d58ba792152e6c512b5, f7a44a5f516545269a3562aad6204c62; no voters:
I20260809 09:31:34.056489 11628 raft_consensus.cc:2781] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 09:31:34.056581 11628 raft_consensus.cc:482] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 09:31:34.056635 11628 raft_consensus.cc:3037] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:34.057453 11628 raft_consensus.cc:504] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a44a5f516545269a3562aad6204c62" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33581 } } peers { permanent_uuid: "28dfb88728f846bc9021c397222b08df" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 44787 } } peers { permanent_uuid: "ab5acafba2a44d58ba792152e6c512b5" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38317 } }
I20260809 09:31:34.057739 11628 leader_election.cc:288] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [CANDIDATE]: Term 1 election: Requested vote from peers f7a44a5f516545269a3562aad6204c62 (127.9.102.3:33581), 28dfb88728f846bc9021c397222b08df (127.9.102.2:44787)
I20260809 09:31:34.058056 11255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a27a00a0ac45d196a9cf96d5669521" candidate_uuid: "ab5acafba2a44d58ba792152e6c512b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28dfb88728f846bc9021c397222b08df"
I20260809 09:31:34.058043 11506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a27a00a0ac45d196a9cf96d5669521" candidate_uuid: "ab5acafba2a44d58ba792152e6c512b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a44a5f516545269a3562aad6204c62"
I20260809 09:31:34.058149 11506 raft_consensus.cc:3037] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:34.058148 11255 raft_consensus.cc:3037] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:34.059173 11255 raft_consensus.cc:2454] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5acafba2a44d58ba792152e6c512b5 in term 1.
I20260809 09:31:34.059173 11506 raft_consensus.cc:2454] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5acafba2a44d58ba792152e6c512b5 in term 1.
I20260809 09:31:34.059433 10934 leader_election.cc:302] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28dfb88728f846bc9021c397222b08df, ab5acafba2a44d58ba792152e6c512b5; no voters:
I20260809 09:31:34.059605 11628 raft_consensus.cc:2781] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:34.059824 11628 raft_consensus.cc:686] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [term 1 LEADER]: Becoming Leader. State: Replica: ab5acafba2a44d58ba792152e6c512b5, State: Running, Role: LEADER
I20260809 09:31:34.060117 11628 consensus_queue.cc:237] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a44a5f516545269a3562aad6204c62" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33581 } } peers { permanent_uuid: "28dfb88728f846bc9021c397222b08df" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 44787 } } peers { permanent_uuid: "ab5acafba2a44d58ba792152e6c512b5" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38317 } }
I20260809 09:31:34.063081 10800 catalog_manager.cc:5495] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5acafba2a44d58ba792152e6c512b5 (127.9.102.1). New cstate: current_term: 1 leader_uuid: "ab5acafba2a44d58ba792152e6c512b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a44a5f516545269a3562aad6204c62" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28dfb88728f846bc9021c397222b08df" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 44787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5acafba2a44d58ba792152e6c512b5" member_type: VOTER last_known_addr { host: "127.9.102.1" port: 38317 } health_report { overall_health: HEALTHY } } }
I20260809 09:31:34.081740 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:34.078361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39270 (local address 127.9.102.1:38317)
0809 09:31:34.078464 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:34.078470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:34.078515 (+ 45us) server_negotiation.cc:407] Connection header received
0809 09:31:34.078686 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:34.078688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:34.078731 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:34.078831 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:31:34.079052 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.079510 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.080398 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:34.080665 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:34.080784 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:34.081054 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:31:34.081392 (+ 338us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:31:34.081512 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:34.081571 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 09:31:34.081612 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 09:31:34.086349 11506 raft_consensus.cc:1261] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62 [term 1 FOLLOWER]: Refusing update from remote peer ab5acafba2a44d58ba792152e6c512b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:31:34.086349 11255 raft_consensus.cc:1261] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df [term 1 FOLLOWER]: Refusing update from remote peer ab5acafba2a44d58ba792152e6c512b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:31:34.086956 11631 consensus_queue.cc:1035] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28dfb88728f846bc9021c397222b08df" member_type: VOTER last_known_addr { host: "127.9.102.2" port: 44787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:31:34.087119 11628 consensus_queue.cc:1035] T 56a27a00a0ac45d196a9cf96d5669521 P ab5acafba2a44d58ba792152e6c512b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7a44a5f516545269a3562aad6204c62" member_type: VOTER last_known_addr { host: "127.9.102.3" port: 33581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:31:34.093851 11636 mvcc.cc:205] Tried to move back new op lower bound from 7316553294171914240 to 7316553294072877056. Current Snapshot: MvccSnapshot[applied={T|T < 7316553294171914240}]
I20260809 09:31:34.095569 11638 mvcc.cc:205] Tried to move back new op lower bound from 7316553294171914240 to 7316553294072877056. Current Snapshot: MvccSnapshot[applied={T|T < 7316553294171914240}]
I20260809 09:31:34.101236 10800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:31:34.101387 10800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 09:31:34.103416 10800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdc0f8f7a85b4005b19fad9522c3fb47: Sending DeleteTablet for 3 replicas of tablet 56a27a00a0ac45d196a9cf96d5669521
I20260809 09:31:34.104017 11470 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a27a00a0ac45d196a9cf96d5669521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:34 UTC) from {username='slave'} at 127.0.0.1:40316
I20260809 09:31:34.104215 11223 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a27a00a0ac45d196a9cf96d5669521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 09:31:34 UTC) from {username='slave'} at 127.0.0.1:39310
I20260809 09:31:34.104462 11654 tablet_replica.cc:330] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62: stopping tablet replica
I20260809 09:31:34.104472 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 10904
I20260809 09:31:34.124722 11655 tablet_replica.cc:330] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df: stopping tablet replica
I20260809 09:31:34.125094 11655 raft_consensus.cc:2229] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:31:34.125450 11655 raft_consensus.cc:2258] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:34.126932 11654 raft_consensus.cc:2229] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 09:31:34.127173 11654 raft_consensus.cc:2258] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 09:31:34.127715 11655 ts_tablet_manager.cc:1905] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:31:34.128371 11654 ts_tablet_manager.cc:1905] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 09:31:34.130367 11654 ts_tablet_manager.cc:1918] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:31:34.130470 11654 log.cc:1196] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-2/wal/wals/56a27a00a0ac45d196a9cf96d5669521
I20260809 09:31:34.130744 11654 ts_tablet_manager.cc:1939] T 56a27a00a0ac45d196a9cf96d5669521 P f7a44a5f516545269a3562aad6204c62: Deleting consensus metadata
I20260809 09:31:34.130950 11655 ts_tablet_manager.cc:1918] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 09:31:34.131050 11655 log.cc:1196] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df: Deleting WAL directory at /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786267646682796-9624-0/minicluster-data/ts-1/wal/wals/56a27a00a0ac45d196a9cf96d5669521
I20260809 09:31:34.131325 11655 ts_tablet_manager.cc:1939] T 56a27a00a0ac45d196a9cf96d5669521 P 28dfb88728f846bc9021c397222b08df: Deleting consensus metadata
I20260809 09:31:34.131793 10776 catalog_manager.cc:4841] TS f7a44a5f516545269a3562aad6204c62 (127.9.102.3:33581): tablet 56a27a00a0ac45d196a9cf96d5669521 (table test-table [id=b8b59664e7f046b894bc0cd929f020ad]) successfully deleted
I20260809 09:31:34.132364 10775 catalog_manager.cc:4841] TS 28dfb88728f846bc9021c397222b08df (127.9.102.2:44787): tablet 56a27a00a0ac45d196a9cf96d5669521 (table test-table [id=b8b59664e7f046b894bc0cd929f020ad]) successfully deleted
W20260809 09:31:34.133241 10775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.102.1:38317 (error 108)
I20260809 09:31:34.133987 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 11124
I20260809 09:31:34.134271 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:34.133953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.1:38317 (local address 127.0.0.1:39278)
0809 09:31:34.134112 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 09:31:34.134184 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.1:38317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 09:31:34.140265 10775 catalog_manager.cc:4584] TS ab5acafba2a44d58ba792152e6c512b5 (127.9.102.1:38317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56a27a00a0ac45d196a9cf96d5669521: Network error: Client connection negotiation failed: client connection to 127.9.102.1:38317: connect: Connection refused (error 111)
I20260809 09:31:34.142450 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 11353
I20260809 09:31:34.150312 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 10748
2026-08-09T09:31:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (938 ms)
[----------] 6 tests from EncryptionPolicyTest (10544 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:34 dist-test-slave-7vjv krb5kdc[11661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:34 dist-test-slave-7vjv krb5kdc[11661](info): set up 2 sockets
Aug 09 09:31:34 dist-test-slave-7vjv krb5kdc[11661](info): commencing operation
krb5kdc: starting...
W20260809 09:31:35.277679 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:35 dist-test-slave-7vjv krb5kdc[11661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 09:31:35.385003 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:35.482988 11681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:35.483193 11681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:35.483228 11681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:35.486917 11681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:35.486966 11681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:35.486984 11681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:35.486999 11681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:35.487012 11681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:35.491966 11681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40727
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:35.492949 11681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:35.493841 11681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:35.501394 11689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:35.501402 11692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:35.501397 11687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:35.501617 11681 server_base.cc:1034] running on GCE node
I20260809 09:31:35.502094 11681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:35.502980 11681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:35.504160 11681 hybrid_clock.cc:648] HybridClock initialized: now 1786267895504145 us; error 27 us; skew 500 ppm
Aug 09 09:31:35 dist-test-slave-7vjv krb5kdc[11661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:35.506575 11681 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 09:31:35.507275 11681 webserver.cc:460] Webserver started at http://127.0.0.1:34049/ using document root <none> and password file <none>
I20260809 09:31:35.507735 11681 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:35.507814 11681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:35.508016 11681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:35.509689 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "33e072b450c34ea09b98efe7023c0fad"
format_stamp: "Formatted at 2026-08-09 09:31:35 on dist-test-slave-7vjv"
server_key: "c3b795e17b7902409a915a3b3aca0a29"
server_key_iv: "776b9578dd60a56615a2b7c09c4f2c5f"
server_key_version: "encryptionkey@0"
I20260809 09:31:35.510182 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "33e072b450c34ea09b98efe7023c0fad"
format_stamp: "Formatted at 2026-08-09 09:31:35 on dist-test-slave-7vjv"
server_key: "c3b795e17b7902409a915a3b3aca0a29"
server_key_iv: "776b9578dd60a56615a2b7c09c4f2c5f"
server_key_version: "encryptionkey@0"
I20260809 09:31:35.513388 11681 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:31:35.515424 11705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:35.516587 11681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:31:35.516728 11681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "33e072b450c34ea09b98efe7023c0fad"
format_stamp: "Formatted at 2026-08-09 09:31:35 on dist-test-slave-7vjv"
server_key: "c3b795e17b7902409a915a3b3aca0a29"
server_key_iv: "776b9578dd60a56615a2b7c09c4f2c5f"
server_key_version: "encryptionkey@0"
I20260809 09:31:35.516825 11681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:35.532663 11681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:35.534897 11681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:35.535077 11681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:35.541157 11681 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40727
I20260809 09:31:35.541221 11789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40727 every 8 connection(s)
I20260809 09:31:35.541949 11681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:35.544490 11791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:35.549108 11791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad: Bootstrap starting.
I20260809 09:31:35.550177 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 11681
I20260809 09:31:35.550307 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:35.550591 9624 external_mini_cluster.cc:1428] Setting key e99dbfcb5153286ab0bb701110e02003
I20260809 09:31:35.551470 11791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:35.552353 11791 log.cc:824] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:35.554263 11791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad: No bootstrap required, opened a new log
I20260809 09:31:35.557907 11791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e072b450c34ea09b98efe7023c0fad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40727 } }
I20260809 09:31:35.558081 11791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:35.558143 11791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e072b450c34ea09b98efe7023c0fad, State: Initialized, Role: FOLLOWER
I20260809 09:31:35.558571 11791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e072b450c34ea09b98efe7023c0fad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40727 } }
I20260809 09:31:35.558717 11791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:35.558777 11791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:35.558877 11791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [term 0 FOLLOWER]: Advancing to term 1
Aug 09 09:31:35 dist-test-slave-7vjv krb5kdc[11661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 09:31:35.559903 11791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e072b450c34ea09b98efe7023c0fad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40727 } }
I20260809 09:31:35.560245 11791 leader_election.cc:302] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33e072b450c34ea09b98efe7023c0fad; no voters:
I20260809 09:31:35.560549 11791 leader_election.cc:288] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:35.560597 11796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:35.560781 11796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [term 1 LEADER]: Becoming Leader. State: Replica: 33e072b450c34ea09b98efe7023c0fad, State: Running, Role: LEADER
I20260809 09:31:35.561074 11796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e072b450c34ea09b98efe7023c0fad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40727 } }
I20260809 09:31:35.561621 11791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:35.562115 11796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e072b450c34ea09b98efe7023c0fad. Latest consensus state: current_term: 1 leader_uuid: "33e072b450c34ea09b98efe7023c0fad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e072b450c34ea09b98efe7023c0fad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40727 } } }
I20260809 09:31:35.562306 11796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:35.562484 11798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33e072b450c34ea09b98efe7023c0fad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e072b450c34ea09b98efe7023c0fad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40727 } } }
I20260809 09:31:35.562729 11808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:35.562834 11798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:35.564534 11794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:35.551642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36617 (local address 127.0.0.1:40727)
0809 09:31:35.552076 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:35.552092 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:35.552147 (+ 55us) server_negotiation.cc:407] Connection header received
0809 09:31:35.552856 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:35.552873 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:35.553219 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:35.553519 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:35.554160 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:35.555565 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:35.556357 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:35.557042 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:35.557185 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:35.559748 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:35.559753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:35.559763 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:35.559787 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:35.561566 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:35.562013 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:35.562018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:35.562024 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:35.562150 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:35.562590 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:35.562593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:35.562596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:35.563038 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:35.563291 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:35.563536 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 09:31:35.563730 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
I20260809 09:31:35.566031 11808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:35.567416 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:35.571058 11808 catalog_manager.cc:1347] Generated new cluster ID: ae36591fa60345959970ea5e2c0da10c
I20260809 09:31:35.571118 11808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:35.576907 11794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:35.569615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33505 (local address 127.0.0.1:40727)
0809 09:31:35.569700 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:35.569703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:35.569910 (+ 207us) server_negotiation.cc:407] Connection header received
0809 09:31:35.569995 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:35.569997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:35.570034 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:35.570107 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:35.570792 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:35.571573 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:35.572419 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:35.572633 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:35.572728 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:35.573848 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:35.573851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:35.573854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:35.573884 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:35.575210 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:35.575676 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:35.575679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:35.575681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:35.575755 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:35.576074 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:35.576077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:35.576079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:35.576282 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:35.576424 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:35.576673 (+ 249us) server_negotiation.cc:299] Negotiation successful
0809 09:31:35.576757 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 09:31:35.581115 11808 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:35.581940 11808 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:35.587317 11808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33e072b450c34ea09b98efe7023c0fad: Generated new TSK 0
I20260809 09:31:35.587803 11808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:35.595245 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 11681
W20260809 09:31:35.608367 11657 connection.cc:535] client connection to 127.0.0.1:40727 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40727): Connection reset by peer (error 104)
2026-08-09T09:31:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:35 dist-test-slave-7vjv krb5kdc[11836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:35 dist-test-slave-7vjv krb5kdc[11836](info): set up 2 sockets
Aug 09 09:31:35 dist-test-slave-7vjv krb5kdc[11836](info): commencing operation
krb5kdc: starting...
W20260809 09:31:36.753247 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:36 dist-test-slave-7vjv krb5kdc[11836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 09:31:36.870492 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:36.973212 11860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:36.973403 11860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:36.973436 11860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:36.977177 11860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:36.977226 11860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:36.977244 11860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:36.977258 11860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:36.977272 11860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:36.981941 11860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35065
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:36.982781 11860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:36.983634 11860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:36.991509 11870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:36.991582 11872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:36.991577 11869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:36.991758 11860 server_base.cc:1034] running on GCE node
I20260809 09:31:36.992389 11860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:36.993361 11860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:36.994544 11860 hybrid_clock.cc:648] HybridClock initialized: now 1786267896994525 us; error 28 us; skew 500 ppm
Aug 09 09:31:36 dist-test-slave-7vjv krb5kdc[11836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:36.997157 11860 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 09:31:36.997929 11860 webserver.cc:460] Webserver started at http://127.0.0.1:37861/ using document root <none> and password file <none>
I20260809 09:31:36.998411 11860 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:36.998495 11860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:36.998728 11860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:37.000628 11860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "8a7d05fbaee44cf18d2cfb3cae193ac8"
format_stamp: "Formatted at 2026-08-09 09:31:36 on dist-test-slave-7vjv"
server_key: "60574958b81db42218895fec0a339247"
server_key_iv: "6113381c180a6395b4c6094eef4bd009"
server_key_version: "encryptionkey@0"
I20260809 09:31:37.001147 11860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "8a7d05fbaee44cf18d2cfb3cae193ac8"
format_stamp: "Formatted at 2026-08-09 09:31:36 on dist-test-slave-7vjv"
server_key: "60574958b81db42218895fec0a339247"
server_key_iv: "6113381c180a6395b4c6094eef4bd009"
server_key_version: "encryptionkey@0"
I20260809 09:31:37.004567 11860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:31:37.006713 11881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:37.007851 11860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:37.007979 11860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "8a7d05fbaee44cf18d2cfb3cae193ac8"
format_stamp: "Formatted at 2026-08-09 09:31:36 on dist-test-slave-7vjv"
server_key: "60574958b81db42218895fec0a339247"
server_key_iv: "6113381c180a6395b4c6094eef4bd009"
server_key_version: "encryptionkey@0"
I20260809 09:31:37.008080 11860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:37.044989 11860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:37.047457 11860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:37.047636 11860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:37.053964 11860 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35065
I20260809 09:31:37.053989 11967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35065 every 8 connection(s)
I20260809 09:31:37.054756 11860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:37.056023 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 11860
I20260809 09:31:37.056193 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:37.056459 9624 external_mini_cluster.cc:1428] Setting key 4a7d637292379e0832a375c62019b86d
I20260809 09:31:37.057474 11968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:37.063014 11968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8: Bootstrap starting.
I20260809 09:31:37.064750 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:37.057713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55349 (local address 127.0.0.1:35065)
0809 09:31:37.058121 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:37.058135 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:37.058167 (+ 32us) server_negotiation.cc:407] Connection header received
0809 09:31:37.058723 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:37.058736 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:37.059027 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:37.059352 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:37.059766 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:37.060818 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:37.061647 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:37.062094 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:37.062248 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:37.063177 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:37.063185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:37.063200 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:37.063557 (+ 357us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:37.063607 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:37.063629 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:37.063768 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:37.063964 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 09:31:37.064135 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":146,"threads_started":1}
I20260809 09:31:37.065377 11968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:37.066113 11968 log.cc:824] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:37.067545 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:37.067991 11968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8: No bootstrap required, opened a new log
I20260809 09:31:37.071744 11968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7d05fbaee44cf18d2cfb3cae193ac8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35065 } }
I20260809 09:31:37.072008 11968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:37.072119 11968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7d05fbaee44cf18d2cfb3cae193ac8, State: Initialized, Role: FOLLOWER
I20260809 09:31:37.072572 11968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7d05fbaee44cf18d2cfb3cae193ac8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35065 } }
I20260809 09:31:37.072734 11968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:37.072798 11968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:37.072894 11968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:37.073522 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:37.069604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45899 (local address 127.0.0.1:35065)
0809 09:31:37.069720 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:37.069724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:37.069933 (+ 209us) server_negotiation.cc:407] Connection header received
0809 09:31:37.070053 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:37.070055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:37.070089 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:37.070156 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:37.070426 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:37.070848 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:37.071878 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:37.072198 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:37.072300 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:37.073099 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:37.073104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:37.073108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:37.073119 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:37.073142 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:37.073155 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:37.073235 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:37.073389 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 09:31:37.073419 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:31:37.073892 11968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7d05fbaee44cf18d2cfb3cae193ac8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35065 } }
I20260809 09:31:37.074203 11968 leader_election.cc:302] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a7d05fbaee44cf18d2cfb3cae193ac8; no voters:
I20260809 09:31:37.074478 11968 leader_election.cc:288] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:37.074568 11987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:37.074802 11987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [term 1 LEADER]: Becoming Leader. State: Replica: 8a7d05fbaee44cf18d2cfb3cae193ac8, State: Running, Role: LEADER
I20260809 09:31:37.075095 11987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7d05fbaee44cf18d2cfb3cae193ac8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35065 } }
I20260809 09:31:37.075558 11968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:37.076676 11989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a7d05fbaee44cf18d2cfb3cae193ac8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7d05fbaee44cf18d2cfb3cae193ac8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35065 } } }
I20260809 09:31:37.076793 11989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:37.077085 11990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7d05fbaee44cf18d2cfb3cae193ac8. Latest consensus state: current_term: 1 leader_uuid: "8a7d05fbaee44cf18d2cfb3cae193ac8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7d05fbaee44cf18d2cfb3cae193ac8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35065 } } }
I20260809 09:31:37.077203 11990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:37.077143 12002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:37.080564 12002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:37.085369 12002 catalog_manager.cc:1347] Generated new cluster ID: 181f0f36db5648708242c1e1d56539a5
I20260809 09:31:37.085440 12002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:37.114988 12002 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:37.115787 12002 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:37.124449 12002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a7d05fbaee44cf18d2cfb3cae193ac8: Generated new TSK 0
I20260809 09:31:37.125074 12002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:37.131426 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 11860
W20260809 09:31:37.150648 11832 connection.cc:535] client connection to 127.0.0.1:35065 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35065): Connection reset by peer (error 104)
2026-08-09T09:31:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1541 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:37 dist-test-slave-7vjv krb5kdc[12019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:37 dist-test-slave-7vjv krb5kdc[12019](info): set up 2 sockets
Aug 09 09:31:37 dist-test-slave-7vjv krb5kdc[12019](info): commencing operation
krb5kdc: starting...
W20260809 09:31:38.304497 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:38 dist-test-slave-7vjv krb5kdc[12019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 09:31:38.413226 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:38.512499 12038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:38.512714 12038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:38.512749 12038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:38.515976 12038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:38.516016 12038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:38.516031 12038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:38.516045 12038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:38.516057 12038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:38.520802 12038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:38.521581 12038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:38.522351 12038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:38.529966 12047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:38.530047 12038 server_base.cc:1034] running on GCE node
W20260809 09:31:38.529917 12050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:38.529979 12046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:38.530756 12038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:38.531715 12038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:38.532938 12038 hybrid_clock.cc:648] HybridClock initialized: now 1786267898532906 us; error 56 us; skew 500 ppm
Aug 09 09:31:38 dist-test-slave-7vjv krb5kdc[12019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:38.535430 12038 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 09:31:38.536209 12038 webserver.cc:460] Webserver started at http://127.0.0.1:40523/ using document root <none> and password file <none>
I20260809 09:31:38.536665 12038 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:38.536720 12038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:38.536864 12038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:38.538658 12038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "3a9335aecad44b1fac15a88b450363aa"
format_stamp: "Formatted at 2026-08-09 09:31:38 on dist-test-slave-7vjv"
server_key: "9784cb45d86f2e532af6cca20f09d024"
server_key_iv: "70466234b3fb37dd8eb9b346a1f0bf70"
server_key_version: "encryptionkey@0"
I20260809 09:31:38.539113 12038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "3a9335aecad44b1fac15a88b450363aa"
format_stamp: "Formatted at 2026-08-09 09:31:38 on dist-test-slave-7vjv"
server_key: "9784cb45d86f2e532af6cca20f09d024"
server_key_iv: "70466234b3fb37dd8eb9b346a1f0bf70"
server_key_version: "encryptionkey@0"
I20260809 09:31:38.542484 12038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260809 09:31:38.544567 12059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:38.545569 12038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:38.545687 12038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "3a9335aecad44b1fac15a88b450363aa"
format_stamp: "Formatted at 2026-08-09 09:31:38 on dist-test-slave-7vjv"
server_key: "9784cb45d86f2e532af6cca20f09d024"
server_key_iv: "70466234b3fb37dd8eb9b346a1f0bf70"
server_key_version: "encryptionkey@0"
I20260809 09:31:38.545778 12038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:38.549773 12038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:38.552265 12038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:38.552428 12038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:38.558394 12038 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37791
I20260809 09:31:38.558405 12154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37791 every 8 connection(s)
I20260809 09:31:38.559104 12038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:38.561437 12155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:38.565977 12155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa: Bootstrap starting.
I20260809 09:31:38.567921 12155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:38.568320 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 12038
I20260809 09:31:38.568451 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:38.568696 12155 log.cc:824] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:38.568828 9624 external_mini_cluster.cc:1428] Setting key bdaee16ff245047900dce6882523fa0e
I20260809 09:31:38.570659 12155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa: No bootstrap required, opened a new log
I20260809 09:31:38.573338 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:38.569814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49585 (local address 127.0.0.1:37791)
0809 09:31:38.570170 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:38.570185 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:38.570226 (+ 41us) server_negotiation.cc:407] Connection header received
0809 09:31:38.570781 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:38.570805 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:38.571056 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:38.571296 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:38.572136 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:38.572148 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:38.572159 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:38.572169 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:38.572215 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:38.572232 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:38.572311 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:38.572443 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 09:31:38.572595 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
I20260809 09:31:38.573848 12155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9335aecad44b1fac15a88b450363aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37791 } }
I20260809 09:31:38.574060 12155 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:38.574117 12155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9335aecad44b1fac15a88b450363aa, State: Initialized, Role: FOLLOWER
I20260809 09:31:38.574532 12155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9335aecad44b1fac15a88b450363aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37791 } }
I20260809 09:31:38.574666 12155 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:38.574729 12155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:38.574837 12155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:38.575691 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:38.575688 12155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9335aecad44b1fac15a88b450363aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37791 } }
I20260809 09:31:38.575995 12155 leader_election.cc:302] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a9335aecad44b1fac15a88b450363aa; no voters:
I20260809 09:31:38.576334 12155 leader_election.cc:288] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:38.576387 12161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:38.576606 12161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [term 1 LEADER]: Becoming Leader. State: Replica: 3a9335aecad44b1fac15a88b450363aa, State: Running, Role: LEADER
I20260809 09:31:38.576934 12161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9335aecad44b1fac15a88b450363aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37791 } }
I20260809 09:31:38.577519 12155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:38.578713 12164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a9335aecad44b1fac15a88b450363aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9335aecad44b1fac15a88b450363aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37791 } } }
I20260809 09:31:38.578713 12166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a9335aecad44b1fac15a88b450363aa. Latest consensus state: current_term: 1 leader_uuid: "3a9335aecad44b1fac15a88b450363aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a9335aecad44b1fac15a88b450363aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37791 } } }
I20260809 09:31:38.578934 12166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:38.578934 12164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:38.579308 12186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:38.580776 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:38.577796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45631 (local address 127.0.0.1:37791)
0809 09:31:38.577891 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:38.577894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:38.578859 (+ 965us) server_negotiation.cc:407] Connection header received
0809 09:31:38.579364 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:38.579368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:38.579403 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:38.579452 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:38.580392 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:38.580394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:38.580396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:38.580405 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:38.580425 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:38.580433 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:38.580502 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:38.580629 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 09:31:38.580656 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260809 09:31:38.582573 12186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:38.587158 12186 catalog_manager.cc:1347] Generated new cluster ID: 73e06c68147b4316b57aadc35497f6ac
I20260809 09:31:38.587218 12186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:38.608656 12186 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:38.608909 12186 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:38.625782 12186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a9335aecad44b1fac15a88b450363aa: Generated new TSK 0
I20260809 09:31:38.626389 12186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:38.635748 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 12038
2026-08-09T09:31:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:38 dist-test-slave-7vjv krb5kdc[12208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:38 dist-test-slave-7vjv krb5kdc[12208](info): set up 2 sockets
Aug 09 09:31:38 dist-test-slave-7vjv krb5kdc[12208](info): commencing operation
krb5kdc: starting...
W20260809 09:31:39.776264 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:39 dist-test-slave-7vjv krb5kdc[12208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 09:31:39.882174 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:39.978824 12233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:39.979015 12233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:39.979054 12233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:39.982584 12233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:39.982626 12233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:39.982643 12233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:39.982658 12233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:39.982672 12233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:39.987107 12233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:39.987931 12233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:39.988754 12233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:39.996369 12244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:39.996381 12238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:39.996433 12240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:39.996552 12233 server_base.cc:1034] running on GCE node
I20260809 09:31:39.997208 12233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:39.998121 12233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:39.999279 12233 hybrid_clock.cc:648] HybridClock initialized: now 1786267899999265 us; error 23 us; skew 500 ppm
Aug 09 09:31:39 dist-test-slave-7vjv krb5kdc[12208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:40.001942 12233 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 09:31:40.002691 12233 webserver.cc:460] Webserver started at http://127.0.0.1:38769/ using document root <none> and password file <none>
I20260809 09:31:40.003156 12233 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:40.003212 12233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:40.003361 12233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:40.005151 12233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "09d091536b2344bcb43ba495e1de887e"
format_stamp: "Formatted at 2026-08-09 09:31:39 on dist-test-slave-7vjv"
server_key: "f418b80aec6d93cad01680e66738831b"
server_key_iv: "32d9f2ac19ea760a899740963769acdf"
server_key_version: "encryptionkey@0"
I20260809 09:31:40.005600 12233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "09d091536b2344bcb43ba495e1de887e"
format_stamp: "Formatted at 2026-08-09 09:31:39 on dist-test-slave-7vjv"
server_key: "f418b80aec6d93cad01680e66738831b"
server_key_iv: "32d9f2ac19ea760a899740963769acdf"
server_key_version: "encryptionkey@0"
I20260809 09:31:40.008834 12233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:31:40.010982 12257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:40.012010 12233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 09:31:40.012164 12233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "09d091536b2344bcb43ba495e1de887e"
format_stamp: "Formatted at 2026-08-09 09:31:39 on dist-test-slave-7vjv"
server_key: "f418b80aec6d93cad01680e66738831b"
server_key_iv: "32d9f2ac19ea760a899740963769acdf"
server_key_version: "encryptionkey@0"
I20260809 09:31:40.012249 12233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:40.029263 12233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:40.031662 12233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:40.031795 12233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:40.038283 12233 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41319
I20260809 09:31:40.038300 12352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41319 every 8 connection(s)
I20260809 09:31:40.039193 12233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:40.041702 12354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:40.046516 12354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e: Bootstrap starting.
I20260809 09:31:40.046948 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 12233
I20260809 09:31:40.047053 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:40.047335 9624 external_mini_cluster.cc:1428] Setting key de329220c647b9e0fa3caacc4d12a931
I20260809 09:31:40.048971 12354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:40.049662 12354 log.cc:824] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:40.051281 12354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e: No bootstrap required, opened a new log
I20260809 09:31:40.054157 12354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d091536b2344bcb43ba495e1de887e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41319 } }
I20260809 09:31:40.054363 12354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:40.054446 12354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d091536b2344bcb43ba495e1de887e, State: Initialized, Role: FOLLOWER
I20260809 09:31:40.054899 12354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d091536b2344bcb43ba495e1de887e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41319 } }
I20260809 09:31:40.055056 12354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:40.055117 12354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Aug 09 09:31:40 dist-test-slave-7vjv krb5kdc[12208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 09:31:40.055235 12354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:40.056205 12354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d091536b2344bcb43ba495e1de887e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41319 } }
I20260809 09:31:40.056607 12354 leader_election.cc:302] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09d091536b2344bcb43ba495e1de887e; no voters:
I20260809 09:31:40.056915 12354 leader_election.cc:288] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:40.057025 12362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:40.057221 12362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [term 1 LEADER]: Becoming Leader. State: Replica: 09d091536b2344bcb43ba495e1de887e, State: Running, Role: LEADER
I20260809 09:31:40.057559 12362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d091536b2344bcb43ba495e1de887e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41319 } }
I20260809 09:31:40.058007 12354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:40.058686 12362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d091536b2344bcb43ba495e1de887e. Latest consensus state: current_term: 1 leader_uuid: "09d091536b2344bcb43ba495e1de887e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d091536b2344bcb43ba495e1de887e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41319 } } }
I20260809 09:31:40.058878 12362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:40.059113 12363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09d091536b2344bcb43ba495e1de887e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d091536b2344bcb43ba495e1de887e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41319 } } }
I20260809 09:31:40.059242 12363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:40.059357 12375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:40.060976 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:40.048500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60873 (local address 127.0.0.1:41319)
0809 09:31:40.048869 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:40.048881 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:40.048912 (+ 31us) server_negotiation.cc:407] Connection header received
0809 09:31:40.049537 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:40.049554 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:40.049901 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:40.050190 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:40.050930 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:40.051836 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:40.052692 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:40.053100 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:40.053251 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:40.055760 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:40.055766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:40.055781 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:40.055809 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:40.057847 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:40.058327 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:40.058332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:40.058337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:40.058470 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:40.058839 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:40.058843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:40.058847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:40.059322 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:40.059537 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:40.059817 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 09:31:40.060149 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
I20260809 09:31:40.062264 12375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:40.063611 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:40.067453 12375 catalog_manager.cc:1347] Generated new cluster ID: 442c92bef88c4904a8350cad99a6bd47
I20260809 09:31:40.067507 12375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:40.072986 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:40.066067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48387 (local address 127.0.0.1:41319)
0809 09:31:40.066171 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:40.066174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:40.066398 (+ 224us) server_negotiation.cc:407] Connection header received
0809 09:31:40.066482 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:40.066484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:40.066537 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:40.066615 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:40.067325 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:40.067832 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:40.068686 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:40.068904 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:40.069004 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:40.070155 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:40.070158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:40.070161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:40.070189 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:40.071423 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:40.071828 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:40.071832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:40.071833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:40.071909 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:40.072216 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:40.072219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:40.072221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:40.072372 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:40.072509 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:40.072727 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 09:31:40.072828 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260809 09:31:40.084482 12375 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:40.085198 12375 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:40.094396 12375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09d091536b2344bcb43ba495e1de887e: Generated new TSK 0
I20260809 09:31:40.094918 12375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:40.099237 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 12233
2026-08-09T09:31:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:40 dist-test-slave-7vjv krb5kdc[12407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:40 dist-test-slave-7vjv krb5kdc[12407](info): set up 2 sockets
Aug 09 09:31:40 dist-test-slave-7vjv krb5kdc[12407](info): commencing operation
krb5kdc: starting...
W20260809 09:31:41.291374 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:41 dist-test-slave-7vjv krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 09:31:41.399433 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:41.499011 12427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:41.499202 12427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:41.499254 12427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:41.502754 12427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:41.502795 12427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:41.502811 12427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:41.502825 12427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:41.502837 12427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:41.507123 12427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:41.507862 12427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:41.508632 12427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:41.515772 12437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:41.515805 12439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:41.515790 12436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:41.516032 12427 server_base.cc:1034] running on GCE node
I20260809 09:31:41.516623 12427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:41.517531 12427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:41.518756 12427 hybrid_clock.cc:648] HybridClock initialized: now 1786267901518744 us; error 32 us; skew 500 ppm
Aug 09 09:31:41 dist-test-slave-7vjv krb5kdc[12407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:41.521425 12427 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 09:31:41.522168 12427 webserver.cc:460] Webserver started at http://127.0.0.1:34657/ using document root <none> and password file <none>
I20260809 09:31:41.522646 12427 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:41.522714 12427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:41.522943 12427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:41.524770 12427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "86f7d8576f534a3d80b98e60668e107e"
format_stamp: "Formatted at 2026-08-09 09:31:41 on dist-test-slave-7vjv"
server_key: "edad898d521d12d6e9edd0a290612b36"
server_key_iv: "e4972a543194877f33a17d4cdb697fe0"
server_key_version: "encryptionkey@0"
I20260809 09:31:41.525279 12427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "86f7d8576f534a3d80b98e60668e107e"
format_stamp: "Formatted at 2026-08-09 09:31:41 on dist-test-slave-7vjv"
server_key: "edad898d521d12d6e9edd0a290612b36"
server_key_iv: "e4972a543194877f33a17d4cdb697fe0"
server_key_version: "encryptionkey@0"
I20260809 09:31:41.528584 12427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 09:31:41.530790 12452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:41.531895 12427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:41.532025 12427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "86f7d8576f534a3d80b98e60668e107e"
format_stamp: "Formatted at 2026-08-09 09:31:41 on dist-test-slave-7vjv"
server_key: "edad898d521d12d6e9edd0a290612b36"
server_key_iv: "e4972a543194877f33a17d4cdb697fe0"
server_key_version: "encryptionkey@0"
I20260809 09:31:41.532145 12427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:41.552471 12427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:41.555133 12427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:41.555307 12427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:41.562314 12427 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38569
I20260809 09:31:41.562341 12544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38569 every 8 connection(s)
I20260809 09:31:41.563169 12427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:41.564700 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 12427
I20260809 09:31:41.564833 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:41.565107 9624 external_mini_cluster.cc:1428] Setting key c787a3a7783738fcc3c7fa88ba4b011c
I20260809 09:31:41.565744 12545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:41.572825 12545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e: Bootstrap starting.
I20260809 09:31:41.575853 12545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:41.576725 12545 log.cc:824] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:41.578756 12545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e: No bootstrap required, opened a new log
I20260809 09:31:41.579577 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:41.566289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35173 (local address 127.0.0.1:38569)
0809 09:31:41.566864 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:41.566885 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:41.572573 (+ 5688us) server_negotiation.cc:407] Connection header received
0809 09:31:41.573331 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:41.573349 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:41.573668 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:41.573962 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:41.574350 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:41.575311 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:41.576401 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:41.576932 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:41.577123 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:41.578007 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:41.578012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:41.578026 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:41.578424 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:41.578472 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:41.578493 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:41.578573 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:41.578738 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 09:31:41.578929 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":168,"threads_started":1}
I20260809 09:31:41.582135 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:41.582063 12545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f7d8576f534a3d80b98e60668e107e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260809 09:31:41.582319 12545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:41.582383 12545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f7d8576f534a3d80b98e60668e107e, State: Initialized, Role: FOLLOWER
I20260809 09:31:41.582793 12545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f7d8576f534a3d80b98e60668e107e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260809 09:31:41.582945 12545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:41.583010 12545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:41.583101 12545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:41.584085 12545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f7d8576f534a3d80b98e60668e107e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260809 09:31:41.584447 12545 leader_election.cc:302] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86f7d8576f534a3d80b98e60668e107e; no voters:
I20260809 09:31:41.584793 12545 leader_election.cc:288] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:41.584864 12564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:41.585052 12564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [term 1 LEADER]: Becoming Leader. State: Replica: 86f7d8576f534a3d80b98e60668e107e, State: Running, Role: LEADER
I20260809 09:31:41.585331 12564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f7d8576f534a3d80b98e60668e107e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260809 09:31:41.585943 12545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:41.586392 12564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86f7d8576f534a3d80b98e60668e107e. Latest consensus state: current_term: 1 leader_uuid: "86f7d8576f534a3d80b98e60668e107e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f7d8576f534a3d80b98e60668e107e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } } }
I20260809 09:31:41.586500 12564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:41.586841 12566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86f7d8576f534a3d80b98e60668e107e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f7d8576f534a3d80b98e60668e107e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } } }
I20260809 09:31:41.586961 12566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:41.586858 12572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:41.588685 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:41.584206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37187 (local address 127.0.0.1:38569)
0809 09:31:41.584430 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:41.584436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:41.584453 (+ 17us) server_negotiation.cc:407] Connection header received
0809 09:31:41.584573 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:41.584576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:41.584611 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:41.584700 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:41.584998 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:41.585609 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:41.586610 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:41.586979 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:41.587093 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:41.588052 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:41.588056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:41.588058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:41.588071 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:41.588130 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:41.588146 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:41.588213 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:41.588518 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 09:31:41.588567 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":138}
I20260809 09:31:41.590328 12572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:41.595443 12572 catalog_manager.cc:1347] Generated new cluster ID: 6cca849dfc9748889b5fa02de301fb2f
I20260809 09:31:41.595497 12572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:41.602669 12572 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:41.603395 12572 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:41.619751 12572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86f7d8576f534a3d80b98e60668e107e: Generated new TSK 0
I20260809 09:31:41.620329 12572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:41.630525 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 12427
2026-08-09T09:31:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1538 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:41 dist-test-slave-7vjv krb5kdc[12598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:41 dist-test-slave-7vjv krb5kdc[12598](info): set up 2 sockets
Aug 09 09:31:41 dist-test-slave-7vjv krb5kdc[12598](info): commencing operation
krb5kdc: starting...
W20260809 09:31:42.799597 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:42 dist-test-slave-7vjv krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 09:31:42.906391 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:43.003918 12627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:43.004208 12627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:43.004288 12627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:43.007939 12627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:43.008018 12627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:43.008081 12627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:43.008181 12627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:43.008258 12627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:43.012909 12627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:43.013784 12627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:43.014729 12627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:43.022450 12643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:43.022449 12637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:43.022652 12638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:43.022785 12627 server_base.cc:1034] running on GCE node
I20260809 09:31:43.023375 12627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:43.024365 12627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:43.025518 12627 hybrid_clock.cc:648] HybridClock initialized: now 1786267903025493 us; error 38 us; skew 500 ppm
Aug 09 09:31:43 dist-test-slave-7vjv krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:43.027951 12627 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 09:31:43.028795 12627 webserver.cc:460] Webserver started at http://127.0.0.1:38351/ using document root <none> and password file <none>
I20260809 09:31:43.029276 12627 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:43.029359 12627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:43.029563 12627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:43.031301 12627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "e37d81a8dade4890b108b8c28eb710c1"
format_stamp: "Formatted at 2026-08-09 09:31:43 on dist-test-slave-7vjv"
server_key: "5d37e29bdafe720985d5c23d4ee99617"
server_key_iv: "011ac04d241bdf54d26c1fa3b739d961"
server_key_version: "encryptionkey@0"
I20260809 09:31:43.031804 12627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "e37d81a8dade4890b108b8c28eb710c1"
format_stamp: "Formatted at 2026-08-09 09:31:43 on dist-test-slave-7vjv"
server_key: "5d37e29bdafe720985d5c23d4ee99617"
server_key_iv: "011ac04d241bdf54d26c1fa3b739d961"
server_key_version: "encryptionkey@0"
I20260809 09:31:43.034933 12627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:31:43.037220 12654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:43.038277 12627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:31:43.038426 12627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "e37d81a8dade4890b108b8c28eb710c1"
format_stamp: "Formatted at 2026-08-09 09:31:43 on dist-test-slave-7vjv"
server_key: "5d37e29bdafe720985d5c23d4ee99617"
server_key_iv: "011ac04d241bdf54d26c1fa3b739d961"
server_key_version: "encryptionkey@0"
I20260809 09:31:43.038532 12627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:43.042943 12627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:43.045470 12627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:43.045667 12627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:43.052102 12627 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39815
I20260809 09:31:43.052160 12735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39815 every 8 connection(s)
I20260809 09:31:43.052882 12627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:43.055377 12736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:43.060288 12736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1: Bootstrap starting.
I20260809 09:31:43.061538 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 12627
I20260809 09:31:43.061659 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:43.061903 9624 external_mini_cluster.cc:1428] Setting key 771dc8b1f0d45823afffe81764c3bc3d
I20260809 09:31:43.062593 12736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:43.063328 12736 log.cc:824] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:43.065147 12736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1: No bootstrap required, opened a new log
I20260809 09:31:43.066324 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:43.062846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34631 (local address 127.0.0.1:39815)
0809 09:31:43.063218 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:43.063230 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:43.063258 (+ 28us) server_negotiation.cc:407] Connection header received
0809 09:31:43.063974 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:43.064002 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:43.064302 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:43.064607 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:43.065296 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:43.065302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:43.065321 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:43.065330 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:43.065375 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:43.065392 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:43.065495 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:43.065612 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 09:31:43.065750 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260809 09:31:43.068783 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:43.068745 12736 raft_consensus.cc:348] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37d81a8dade4890b108b8c28eb710c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39815 } }
I20260809 09:31:43.068972 12736 raft_consensus.cc:374] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:43.069023 12736 raft_consensus.cc:729] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37d81a8dade4890b108b8c28eb710c1, State: Initialized, Role: FOLLOWER
I20260809 09:31:43.069351 12736 consensus_queue.cc:260] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37d81a8dade4890b108b8c28eb710c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39815 } }
I20260809 09:31:43.069478 12736 raft_consensus.cc:388] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:43.069552 12736 raft_consensus.cc:482] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:43.069636 12736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:43.070636 12736 raft_consensus.cc:504] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37d81a8dade4890b108b8c28eb710c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39815 } }
I20260809 09:31:43.070998 12736 leader_election.cc:302] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e37d81a8dade4890b108b8c28eb710c1; no voters:
I20260809 09:31:43.071369 12736 leader_election.cc:288] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:43.071446 12755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:43.071708 12755 raft_consensus.cc:686] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [term 1 LEADER]: Becoming Leader. State: Replica: e37d81a8dade4890b108b8c28eb710c1, State: Running, Role: LEADER
I20260809 09:31:43.072016 12755 consensus_queue.cc:237] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37d81a8dade4890b108b8c28eb710c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39815 } }
I20260809 09:31:43.072537 12736 sys_catalog.cc:564] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:43.072554 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:43.070909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58911 (local address 127.0.0.1:39815)
0809 09:31:43.071019 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:43.071022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:43.071231 (+ 209us) server_negotiation.cc:407] Connection header received
0809 09:31:43.071392 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:43.071396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:43.071426 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:43.071508 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:43.072221 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:43.072224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:43.072226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:43.072236 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:43.072257 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:43.072267 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:43.072342 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:43.072435 (+ 93us) server_negotiation.cc:299] Negotiation successful
0809 09:31:43.072464 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 09:31:43.073108 12755 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e37d81a8dade4890b108b8c28eb710c1. Latest consensus state: current_term: 1 leader_uuid: "e37d81a8dade4890b108b8c28eb710c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37d81a8dade4890b108b8c28eb710c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39815 } } }
I20260809 09:31:43.073285 12755 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:43.073714 12772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:43.073740 12760 sys_catalog.cc:455] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e37d81a8dade4890b108b8c28eb710c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e37d81a8dade4890b108b8c28eb710c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39815 } } }
I20260809 09:31:43.073825 12760 sys_catalog.cc:458] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:43.076380 12772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:43.081106 12772 catalog_manager.cc:1347] Generated new cluster ID: 7a252e9b2880443ab68d4c92125a1d5e
I20260809 09:31:43.081163 12772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:43.090327 12772 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:43.090575 12772 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:43.094735 12772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e37d81a8dade4890b108b8c28eb710c1: Generated new TSK 0
I20260809 09:31:43.095322 12772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:43.098794 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 12627
2026-08-09T09:31:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:43 dist-test-slave-7vjv krb5kdc[12797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:43 dist-test-slave-7vjv krb5kdc[12797](info): set up 2 sockets
Aug 09 09:31:43 dist-test-slave-7vjv krb5kdc[12797](info): commencing operation
krb5kdc: starting...
W20260809 09:31:44.268785 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:44 dist-test-slave-7vjv krb5kdc[12797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 09:31:44.379233 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:44.480667 12816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:44.480944 12816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:44.481024 12816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:44.484881 12816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:44.484956 12816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:44.484988 12816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:44.485054 12816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:44.485090 12816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:44.489967 12816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:44.490775 12816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:44.491647 12816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:44.498973 12823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:44.498992 12822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:44.498978 12830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:44.499253 12816 server_base.cc:1034] running on GCE node
I20260809 09:31:44.499826 12816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:44.500810 12816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:44.501989 12816 hybrid_clock.cc:648] HybridClock initialized: now 1786267904501965 us; error 44 us; skew 500 ppm
Aug 09 09:31:44 dist-test-slave-7vjv krb5kdc[12797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:44.506615 12816 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 09:31:44.507300 12816 webserver.cc:460] Webserver started at http://127.0.0.1:42725/ using document root <none> and password file <none>
I20260809 09:31:44.507722 12816 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:44.507772 12816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:44.507903 12816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:44.509552 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "c818e806c7534d97968e96736241b111"
format_stamp: "Formatted at 2026-08-09 09:31:44 on dist-test-slave-7vjv"
server_key: "e88b175d0fb068b13d22b7dd6e412939"
server_key_iv: "16c26d29ea5f64907bff9339473ebce9"
server_key_version: "encryptionkey@0"
I20260809 09:31:44.509985 12816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "c818e806c7534d97968e96736241b111"
format_stamp: "Formatted at 2026-08-09 09:31:44 on dist-test-slave-7vjv"
server_key: "e88b175d0fb068b13d22b7dd6e412939"
server_key_iv: "16c26d29ea5f64907bff9339473ebce9"
server_key_version: "encryptionkey@0"
I20260809 09:31:44.513126 12816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:31:44.515115 12840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:44.516191 12816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260809 09:31:44.516290 12816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "c818e806c7534d97968e96736241b111"
format_stamp: "Formatted at 2026-08-09 09:31:44 on dist-test-slave-7vjv"
server_key: "e88b175d0fb068b13d22b7dd6e412939"
server_key_iv: "16c26d29ea5f64907bff9339473ebce9"
server_key_version: "encryptionkey@0"
I20260809 09:31:44.516362 12816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:44.542977 12816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:44.545346 12816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:44.545470 12816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:44.551959 12816 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37827
I20260809 09:31:44.551973 12936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37827 every 8 connection(s)
I20260809 09:31:44.552786 12816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:44.554895 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 12816
I20260809 09:31:44.555048 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:44.555177 12939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:44.555359 9624 external_mini_cluster.cc:1428] Setting key c2a13d77259a429b17089df7446b0313
I20260809 09:31:44.560518 12939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111: Bootstrap starting.
I20260809 09:31:44.562558 12939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:44.563230 12939 log.cc:824] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111: Log is configured to *not* fsync() on all Append() calls
Aug 09 09:31:44 dist-test-slave-7vjv krb5kdc[12797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 09:31:44.565035 12939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111: No bootstrap required, opened a new log
I20260809 09:31:44.568058 12939 raft_consensus.cc:348] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c818e806c7534d97968e96736241b111" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37827 } }
I20260809 09:31:44.568286 12939 raft_consensus.cc:374] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:44.568361 12939 raft_consensus.cc:729] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c818e806c7534d97968e96736241b111, State: Initialized, Role: FOLLOWER
I20260809 09:31:44.568696 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:44.556625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:46467 (local address 127.0.0.1:37827)
0809 09:31:44.557029 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:44.557045 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:44.557075 (+ 30us) server_negotiation.cc:407] Connection header received
0809 09:31:44.557849 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:44.557866 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:44.558140 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:44.558448 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:44.559254 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:44.560212 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:44.561106 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:44.561528 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:44.561710 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:44.564276 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:44.564282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:44.564293 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:44.564318 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:44.565852 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:44.566411 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:44.566417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:44.566429 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:44.566535 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:44.566893 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:44.566896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:44.566898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:44.567218 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:44.567520 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:44.567778 (+ 258us) server_negotiation.cc:299] Negotiation successful
0809 09:31:44.567966 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
I20260809 09:31:44.568773 12939 consensus_queue.cc:260] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c818e806c7534d97968e96736241b111" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37827 } }
I20260809 09:31:44.568924 12939 raft_consensus.cc:388] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:44.568980 12939 raft_consensus.cc:482] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:44.569068 12939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:44.569759 12939 raft_consensus.cc:504] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c818e806c7534d97968e96736241b111" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37827 } }
I20260809 09:31:44.570086 12939 leader_election.cc:302] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c818e806c7534d97968e96736241b111; no voters:
I20260809 09:31:44.570360 12939 leader_election.cc:288] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:44.570497 12948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:44.570700 12948 raft_consensus.cc:686] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [term 1 LEADER]: Becoming Leader. State: Replica: c818e806c7534d97968e96736241b111, State: Running, Role: LEADER
I20260809 09:31:44.571051 12948 consensus_queue.cc:237] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c818e806c7534d97968e96736241b111" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37827 } }
I20260809 09:31:44.571429 12939 sys_catalog.cc:564] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:44.571448 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:44.572528 12950 sys_catalog.cc:455] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c818e806c7534d97968e96736241b111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c818e806c7534d97968e96736241b111" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37827 } } }
I20260809 09:31:44.572652 12950 sys_catalog.cc:458] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:44.572881 12950 sys_catalog.cc:455] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c818e806c7534d97968e96736241b111. Latest consensus state: current_term: 1 leader_uuid: "c818e806c7534d97968e96736241b111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c818e806c7534d97968e96736241b111" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37827 } } }
I20260809 09:31:44.572969 12950 sys_catalog.cc:458] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:44.573352 12970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:44.575680 12970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:44.581259 12970 catalog_manager.cc:1347] Generated new cluster ID: 44f82ce8ce994990bf5dc00b4f221f67
I20260809 09:31:44.581355 12970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:44.584681 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:44.576582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:35373 (local address 127.0.0.1:37827)
0809 09:31:44.576680 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:44.576684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:44.576963 (+ 279us) server_negotiation.cc:407] Connection header received
0809 09:31:44.577068 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:44.577072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:44.577117 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:44.577196 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:44.577905 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:44.578341 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:44.579115 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:44.579476 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:44.579602 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:44.581565 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:44.581569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:44.581571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:44.581594 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:44.582927 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:44.583389 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:44.583392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:44.583394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:44.583474 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:44.583800 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:44.583804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:44.583805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:44.583964 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:44.584156 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:44.584413 (+ 257us) server_negotiation.cc:299] Negotiation successful
0809 09:31:44.584511 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 09:31:44.598708 12970 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:44.599421 12970 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:44.609428 12970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c818e806c7534d97968e96736241b111: Generated new TSK 0
I20260809 09:31:44.609910 12970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:44.617189 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 12816
W20260809 09:31:44.636152 12794 connection.cc:535] client connection to 127.0.0.1:37827 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37827): Connection reset by peer (error 104)
2026-08-09T09:31:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1529 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:44 dist-test-slave-7vjv krb5kdc[12987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:44 dist-test-slave-7vjv krb5kdc[12987](info): set up 2 sockets
Aug 09 09:31:44 dist-test-slave-7vjv krb5kdc[12987](info): commencing operation
krb5kdc: starting...
W20260809 09:31:45.769176 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:45 dist-test-slave-7vjv krb5kdc[12987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 09:31:45.876585 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:45.973945 13013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:45.974152 13013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:45.974185 13013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:45.977643 13013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:45.977684 13013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:45.977700 13013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:45.977715 13013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:45.977726 13013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:45.982261 13013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:45.983000 13013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:45.983811 13013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:45.990871 13020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:45.990881 13026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:45.990888 13021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:45.991266 13013 server_base.cc:1034] running on GCE node
I20260809 09:31:45.991782 13013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:45.992712 13013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:45.993911 13013 hybrid_clock.cc:648] HybridClock initialized: now 1786267905993895 us; error 27 us; skew 500 ppm
Aug 09 09:31:45 dist-test-slave-7vjv krb5kdc[12987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:45.996316 13013 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 09:31:45.997083 13013 webserver.cc:460] Webserver started at http://127.0.0.1:42289/ using document root <none> and password file <none>
I20260809 09:31:45.997545 13013 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:45.997627 13013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:45.997824 13013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:45.999497 13013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "ee18bdacc1554c51a17c44cf348313e2"
format_stamp: "Formatted at 2026-08-09 09:31:45 on dist-test-slave-7vjv"
server_key: "35b67dde5773a8d426f8c4cfb405f864"
server_key_iv: "0ed79ffdb2f46bafc7de97c579b3c7b3"
server_key_version: "encryptionkey@0"
I20260809 09:31:46.000042 13013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "ee18bdacc1554c51a17c44cf348313e2"
format_stamp: "Formatted at 2026-08-09 09:31:45 on dist-test-slave-7vjv"
server_key: "35b67dde5773a8d426f8c4cfb405f864"
server_key_iv: "0ed79ffdb2f46bafc7de97c579b3c7b3"
server_key_version: "encryptionkey@0"
I20260809 09:31:46.003640 13013 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 09:31:46.005937 13039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:46.007041 13013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:46.007241 13013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "ee18bdacc1554c51a17c44cf348313e2"
format_stamp: "Formatted at 2026-08-09 09:31:45 on dist-test-slave-7vjv"
server_key: "35b67dde5773a8d426f8c4cfb405f864"
server_key_iv: "0ed79ffdb2f46bafc7de97c579b3c7b3"
server_key_version: "encryptionkey@0"
I20260809 09:31:46.007369 13013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:46.024818 13013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:46.027690 13013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:46.027866 13013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:46.034045 13013 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37179
I20260809 09:31:46.034067 13136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37179 every 8 connection(s)
I20260809 09:31:46.034837 13013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:46.037470 13140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:46.041649 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 13013
I20260809 09:31:46.041783 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:46.042109 9624 external_mini_cluster.cc:1428] Setting key 1f9c57f47d5982fe0cd2eee59e2fd24e
I20260809 09:31:46.042577 13140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2: Bootstrap starting.
I20260809 09:31:46.044762 13140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:46.045600 13140 log.cc:824] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:46.047648 13140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2: No bootstrap required, opened a new log
I20260809 09:31:46.050510 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:46.043276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:49905 (local address 127.0.0.1:37179)
0809 09:31:46.043645 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:46.043658 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:46.043690 (+ 32us) server_negotiation.cc:407] Connection header received
0809 09:31:46.044308 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:46.044325 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:46.044622 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:46.044911 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:46.045330 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:46.046283 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:46.047439 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:46.047948 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:46.048065 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:46.048911 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:46.048915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:46.048926 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:46.049340 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:46.049389 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:46.049409 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:46.049502 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:46.049750 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 09:31:46.049904 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":124,"threads_started":1}
I20260809 09:31:46.051344 13140 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee18bdacc1554c51a17c44cf348313e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37179 } }
I20260809 09:31:46.051617 13140 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:46.051674 13140 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee18bdacc1554c51a17c44cf348313e2, State: Initialized, Role: FOLLOWER
I20260809 09:31:46.052074 13140 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee18bdacc1554c51a17c44cf348313e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37179 } }
I20260809 09:31:46.052229 13140 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:46.052345 13140 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:46.052440 13140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:46.053344 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:46.053378 13140 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee18bdacc1554c51a17c44cf348313e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37179 } }
I20260809 09:31:46.053710 13140 leader_election.cc:302] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee18bdacc1554c51a17c44cf348313e2; no voters:
I20260809 09:31:46.054025 13140 leader_election.cc:288] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:46.054091 13148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:46.054293 13148 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [term 1 LEADER]: Becoming Leader. State: Replica: ee18bdacc1554c51a17c44cf348313e2, State: Running, Role: LEADER
I20260809 09:31:46.054625 13148 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee18bdacc1554c51a17c44cf348313e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37179 } }
I20260809 09:31:46.055204 13140 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:46.056286 13149 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee18bdacc1554c51a17c44cf348313e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee18bdacc1554c51a17c44cf348313e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37179 } } }
I20260809 09:31:46.056414 13149 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:46.056653 13149 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee18bdacc1554c51a17c44cf348313e2. Latest consensus state: current_term: 1 leader_uuid: "ee18bdacc1554c51a17c44cf348313e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee18bdacc1554c51a17c44cf348313e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37179 } } }
I20260809 09:31:46.056726 13149 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:46.056815 13170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:46.059736 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:46.055424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:33315 (local address 127.0.0.1:37179)
0809 09:31:46.055551 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:46.055555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:46.055748 (+ 193us) server_negotiation.cc:407] Connection header received
0809 09:31:46.055882 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:46.055885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:46.055924 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:46.055995 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:46.056488 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:46.057211 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:46.058043 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:46.058344 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:46.058444 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:46.059273 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:46.059276 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:46.059279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:46.059291 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:46.059315 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:46.059327 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:46.059418 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:46.059566 (+ 148us) server_negotiation.cc:299] Negotiation successful
0809 09:31:46.059609 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 09:31:46.060599 13170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:46.065376 13170 catalog_manager.cc:1347] Generated new cluster ID: 0370be8f4773412e9d3e1762dc567cd8
I20260809 09:31:46.065430 13170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:46.085964 13170 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:46.086961 13170 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:46.092388 13170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee18bdacc1554c51a17c44cf348313e2: Generated new TSK 0
I20260809 09:31:46.092994 13170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:46.095849 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 13013
W20260809 09:31:46.110441 12984 connection.cc:535] client connection to 127.0.0.1:37179 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37179): Connection reset by peer (error 104)
2026-08-09T09:31:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1474 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:46 dist-test-slave-7vjv krb5kdc[13189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:46 dist-test-slave-7vjv krb5kdc[13189](info): set up 2 sockets
Aug 09 09:31:46 dist-test-slave-7vjv krb5kdc[13189](info): commencing operation
krb5kdc: starting...
W20260809 09:31:47.302428 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:47 dist-test-slave-7vjv krb5kdc[13189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 09:31:47.412168 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:47.508497 13212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:47.508695 13212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:47.508728 13212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:47.512293 13212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:47.512342 13212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:47.512362 13212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:47.512377 13212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:47.512391 13212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:47.516834 13212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:47.517618 13212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:47.518412 13212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:47.525652 13220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:47.525877 13212 server_base.cc:1034] running on GCE node
W20260809 09:31:47.525632 13223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:47.525671 13228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:47.526676 13212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:47.527544 13212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:47.528731 13212 hybrid_clock.cc:648] HybridClock initialized: now 1786267907528715 us; error 26 us; skew 500 ppm
Aug 09 09:31:47 dist-test-slave-7vjv krb5kdc[13189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:47.531286 13212 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 09:31:47.532075 13212 webserver.cc:460] Webserver started at http://127.0.0.1:32987/ using document root <none> and password file <none>
I20260809 09:31:47.532629 13212 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:47.532716 13212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:47.532928 13212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:47.534720 13212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "32eebc5623bc4996b8e8f73bb6843082"
format_stamp: "Formatted at 2026-08-09 09:31:47 on dist-test-slave-7vjv"
server_key: "c49eb5070358312b4d69b338d28d64af"
server_key_iv: "d600031e8ec1ead5c5d2c20c9aebaa6a"
server_key_version: "encryptionkey@0"
I20260809 09:31:47.535259 13212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "32eebc5623bc4996b8e8f73bb6843082"
format_stamp: "Formatted at 2026-08-09 09:31:47 on dist-test-slave-7vjv"
server_key: "c49eb5070358312b4d69b338d28d64af"
server_key_iv: "d600031e8ec1ead5c5d2c20c9aebaa6a"
server_key_version: "encryptionkey@0"
I20260809 09:31:47.538446 13212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260809 09:31:47.540565 13237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:47.541556 13212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260809 09:31:47.541690 13212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "32eebc5623bc4996b8e8f73bb6843082"
format_stamp: "Formatted at 2026-08-09 09:31:47 on dist-test-slave-7vjv"
server_key: "c49eb5070358312b4d69b338d28d64af"
server_key_iv: "d600031e8ec1ead5c5d2c20c9aebaa6a"
server_key_version: "encryptionkey@0"
I20260809 09:31:47.541800 13212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:47.545683 13212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:47.548573 13212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:47.548753 13212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:47.554960 13212 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43173
I20260809 09:31:47.554977 13327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43173 every 8 connection(s)
I20260809 09:31:47.555703 13212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:47.557332 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 13212
I20260809 09:31:47.557477 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:47.557834 9624 external_mini_cluster.cc:1428] Setting key eeb49f2d29721b0167439912f8a74e85
I20260809 09:31:47.558844 13328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:47.562862 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:47.559078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:45101 (local address 127.0.0.1:43173)
0809 09:31:47.559420 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:47.559441 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:47.559471 (+ 30us) server_negotiation.cc:407] Connection header received
0809 09:31:47.560014 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:47.560028 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:47.560344 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:47.560591 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:47.561408 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:47.561414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:47.561431 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:47.561444 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:47.561508 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:47.561537 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:47.561670 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:47.561958 (+ 288us) server_negotiation.cc:299] Negotiation successful
0809 09:31:47.562100 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":120,"threads_started":1}
I20260809 09:31:47.565012 13328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082: Bootstrap starting.
I20260809 09:31:47.565307 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:47.566987 13328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:47.567883 13328 log.cc:824] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:47.568470 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:47.567085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:52335 (local address 127.0.0.1:43173)
0809 09:31:47.567201 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:47.567207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:47.567317 (+ 110us) server_negotiation.cc:407] Connection header received
0809 09:31:47.567378 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:47.567381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:47.567411 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:47.567491 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:47.568076 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:47.568079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:47.568081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:47.568109 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:47.568132 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:47.568142 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:47.568233 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:47.568333 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 09:31:47.568366 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 09:31:47.569621 13328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082: No bootstrap required, opened a new log
I20260809 09:31:47.572444 13328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32eebc5623bc4996b8e8f73bb6843082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43173 } }
I20260809 09:31:47.572643 13328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:47.572711 13328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32eebc5623bc4996b8e8f73bb6843082, State: Initialized, Role: FOLLOWER
I20260809 09:31:47.573109 13328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32eebc5623bc4996b8e8f73bb6843082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43173 } }
I20260809 09:31:47.573267 13328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:47.573348 13328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:47.573436 13328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:47.574213 13328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32eebc5623bc4996b8e8f73bb6843082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43173 } }
I20260809 09:31:47.574518 13328 leader_election.cc:302] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32eebc5623bc4996b8e8f73bb6843082; no voters:
I20260809 09:31:47.574786 13328 leader_election.cc:288] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:47.574889 13343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:47.575081 13343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [term 1 LEADER]: Becoming Leader. State: Replica: 32eebc5623bc4996b8e8f73bb6843082, State: Running, Role: LEADER
I20260809 09:31:47.575443 13343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32eebc5623bc4996b8e8f73bb6843082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43173 } }
I20260809 09:31:47.575811 13328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:47.576929 13344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32eebc5623bc4996b8e8f73bb6843082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32eebc5623bc4996b8e8f73bb6843082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43173 } } }
I20260809 09:31:47.577044 13344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:47.577400 13348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32eebc5623bc4996b8e8f73bb6843082. Latest consensus state: current_term: 1 leader_uuid: "32eebc5623bc4996b8e8f73bb6843082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32eebc5623bc4996b8e8f73bb6843082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43173 } } }
I20260809 09:31:47.577445 13359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:47.577509 13348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:47.580485 13359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:47.585176 13359 catalog_manager.cc:1347] Generated new cluster ID: 3edc836ff7d3444289f18411750ead79
I20260809 09:31:47.585233 13359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:47.595018 13359 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:47.595263 13359 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:47.601068 13359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32eebc5623bc4996b8e8f73bb6843082: Generated new TSK 0
I20260809 09:31:47.601691 13359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:47.604225 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 13212
2026-08-09T09:31:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1504 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:47 dist-test-slave-7vjv krb5kdc[13376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:47 dist-test-slave-7vjv krb5kdc[13376](info): set up 2 sockets
Aug 09 09:31:47 dist-test-slave-7vjv krb5kdc[13376](info): commencing operation
krb5kdc: starting...
W20260809 09:31:48.768582 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:48 dist-test-slave-7vjv krb5kdc[13376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 09:31:48.879829 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:48.982074 13403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:48.982266 13403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:48.982317 13403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:48.985806 13403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:48.985845 13403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:48.985879 13403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:48.985893 13403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:48.985908 13403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:48.990347 13403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:48.991070 13403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:48.991827 13403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:48.999181 13419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:48.999320 13403 server_base.cc:1034] running on GCE node
W20260809 09:31:48.999169 13412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:48.999169 13409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:48.999997 13403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:49.000926 13403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:49.002108 13403 hybrid_clock.cc:648] HybridClock initialized: now 1786267909002083 us; error 36 us; skew 500 ppm
Aug 09 09:31:49 dist-test-slave-7vjv krb5kdc[13376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:49.004765 13403 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 09:31:49.005545 13403 webserver.cc:460] Webserver started at http://127.0.0.1:35933/ using document root <none> and password file <none>
I20260809 09:31:49.006057 13403 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:49.006145 13403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:49.006350 13403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:49.008208 13403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "3c8ca684807146db988db428a27d5176"
format_stamp: "Formatted at 2026-08-09 09:31:49 on dist-test-slave-7vjv"
server_key: "c4bdb228ec52908a3c2d7a70e001c5cf"
server_key_iv: "2d05546127d38da37a52575258ce61c6"
server_key_version: "encryptionkey@0"
I20260809 09:31:49.008739 13403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "3c8ca684807146db988db428a27d5176"
format_stamp: "Formatted at 2026-08-09 09:31:49 on dist-test-slave-7vjv"
server_key: "c4bdb228ec52908a3c2d7a70e001c5cf"
server_key_iv: "2d05546127d38da37a52575258ce61c6"
server_key_version: "encryptionkey@0"
I20260809 09:31:49.011883 13403 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:31:49.014115 13428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:49.015156 13403 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:49.015282 13403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "3c8ca684807146db988db428a27d5176"
format_stamp: "Formatted at 2026-08-09 09:31:49 on dist-test-slave-7vjv"
server_key: "c4bdb228ec52908a3c2d7a70e001c5cf"
server_key_iv: "2d05546127d38da37a52575258ce61c6"
server_key_version: "encryptionkey@0"
I20260809 09:31:49.015380 13403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:49.034207 13403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:49.036738 13403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:49.036914 13403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:49.043143 13403 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45973
I20260809 09:31:49.043157 13529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45973 every 8 connection(s)
I20260809 09:31:49.043941 13403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:49.045415 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 13403
I20260809 09:31:49.045545 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:49.045825 9624 external_mini_cluster.cc:1428] Setting key ee979802c678baa01607505aca2befe5
I20260809 09:31:49.046878 13531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:49.052840 13531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176: Bootstrap starting.
Aug 09 09:31:49 dist-test-slave-7vjv krb5kdc[13376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 09:31:49.055190 13531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:49.056077 13531 log.cc:824] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:49.058339 13531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176: No bootstrap required, opened a new log
I20260809 09:31:49.059516 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:49.047027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:37519 (local address 127.0.0.1:45973)
0809 09:31:49.047393 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:49.047406 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:49.047435 (+ 29us) server_negotiation.cc:407] Connection header received
0809 09:31:49.047983 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:49.047996 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:49.048346 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:49.048612 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:49.049485 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:49.050419 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:49.051465 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:49.051855 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:49.051958 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:49.054537 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:49.054543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:49.054556 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:49.054583 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:49.056472 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:49.057123 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:49.057129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:49.057134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:49.057257 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:49.057561 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:49.057565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:49.057569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:49.058057 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:49.058259 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:49.058472 (+ 213us) server_negotiation.cc:299] Negotiation successful
0809 09:31:49.058693 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":123,"threads_started":1}
I20260809 09:31:49.062381 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:49.062440 13531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8ca684807146db988db428a27d5176" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } }
I20260809 09:31:49.062670 13531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:49.062745 13531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8ca684807146db988db428a27d5176, State: Initialized, Role: FOLLOWER
I20260809 09:31:49.063150 13531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8ca684807146db988db428a27d5176" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } }
I20260809 09:31:49.063278 13531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:49.063352 13531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:49.063441 13531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:49.064394 13531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8ca684807146db988db428a27d5176" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } }
I20260809 09:31:49.064805 13531 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c8ca684807146db988db428a27d5176; no voters:
I20260809 09:31:49.065136 13531 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:49.065220 13549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:49.065409 13549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [term 1 LEADER]: Becoming Leader. State: Replica: 3c8ca684807146db988db428a27d5176, State: Running, Role: LEADER
I20260809 09:31:49.065725 13549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8ca684807146db988db428a27d5176" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } }
I20260809 09:31:49.066210 13531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:49.067174 13550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c8ca684807146db988db428a27d5176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8ca684807146db988db428a27d5176" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } } }
I20260809 09:31:49.067293 13550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:49.067662 13551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c8ca684807146db988db428a27d5176. Latest consensus state: current_term: 1 leader_uuid: "3c8ca684807146db988db428a27d5176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8ca684807146db988db428a27d5176" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } } }
I20260809 09:31:49.067804 13551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176 [sys.catalog]: This master's current role is: LEADER
W20260809 09:31:49.070912 13582 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:31:49.070990 13582 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:31:49.071076 13563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:49.071815 13563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:49.072414 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:49.064431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:59747 (local address 127.0.0.1:45973)
0809 09:31:49.064529 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:49.064532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:49.064666 (+ 134us) server_negotiation.cc:407] Connection header received
0809 09:31:49.064758 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:49.064760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:49.064791 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:49.064847 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:49.065632 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:49.066299 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:49.067492 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:49.067847 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:49.067951 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:49.069383 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:49.069386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:49.069388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:49.069410 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:49.070687 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:49.071200 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:49.071202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:49.071203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:49.071268 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:49.071582 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:49.071585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:49.071587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:49.071734 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:49.071867 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:49.072163 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 09:31:49.072250 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260809 09:31:49.076785 13563 catalog_manager.cc:1347] Generated new cluster ID: 1aab16279a6c47b3a0e6dd2dc1aaab35
I20260809 09:31:49.076853 13563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:49.081203 13563 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:49.081900 13563 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:49.085457 13563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c8ca684807146db988db428a27d5176: Generated new TSK 0
I20260809 09:31:49.086015 13563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:49.090052 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 13403
W20260809 09:31:49.106832 13372 connection.cc:535] client connection to 127.0.0.1:45973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45973): Connection reset by peer (error 104)
2026-08-09T09:31:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1491 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:49 dist-test-slave-7vjv krb5kdc[13592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:49 dist-test-slave-7vjv krb5kdc[13592](info): set up 2 sockets
Aug 09 09:31:49 dist-test-slave-7vjv krb5kdc[13592](info): commencing operation
krb5kdc: starting...
W20260809 09:31:50.278997 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:50 dist-test-slave-7vjv krb5kdc[13592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 09:31:50.388871 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:50.488718 13609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:50.488935 13609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:50.488972 13609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:50.492477 13609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:50.492518 13609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:50.492535 13609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:50.492549 13609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:50.492563 13609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:50.496908 13609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39789
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:50.497664 13609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:50.498466 13609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:50.506073 13622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:50.506086 13621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:50.506091 13625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:50.506493 13609 server_base.cc:1034] running on GCE node
I20260809 09:31:50.506994 13609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:50.507882 13609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:50.509032 13609 hybrid_clock.cc:648] HybridClock initialized: now 1786267910509014 us; error 29 us; skew 500 ppm
Aug 09 09:31:50 dist-test-slave-7vjv krb5kdc[13592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:50.515544 13609 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 09:31:50.516359 13609 webserver.cc:460] Webserver started at http://127.0.0.1:42313/ using document root <none> and password file <none>
I20260809 09:31:50.516909 13609 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:50.517000 13609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:50.517217 13609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:50.519089 13609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "e783d66c97d04b93b0539515186f00cb"
format_stamp: "Formatted at 2026-08-09 09:31:50 on dist-test-slave-7vjv"
server_key: "ed473962cf62262fabc4b5206451021a"
server_key_iv: "2725823248099a600aa2af5ca21153d0"
server_key_version: "encryptionkey@0"
I20260809 09:31:50.519718 13609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "e783d66c97d04b93b0539515186f00cb"
format_stamp: "Formatted at 2026-08-09 09:31:50 on dist-test-slave-7vjv"
server_key: "ed473962cf62262fabc4b5206451021a"
server_key_iv: "2725823248099a600aa2af5ca21153d0"
server_key_version: "encryptionkey@0"
I20260809 09:31:50.523051 13609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 09:31:50.525382 13635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:50.526496 13609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:31:50.526638 13609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "e783d66c97d04b93b0539515186f00cb"
format_stamp: "Formatted at 2026-08-09 09:31:50 on dist-test-slave-7vjv"
server_key: "ed473962cf62262fabc4b5206451021a"
server_key_iv: "2725823248099a600aa2af5ca21153d0"
server_key_version: "encryptionkey@0"
I20260809 09:31:50.526738 13609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:50.542948 13609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:50.545396 13609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:50.545586 13609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:50.551707 13609 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39789
I20260809 09:31:50.551779 13730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39789 every 8 connection(s)
I20260809 09:31:50.552572 13609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:50.554283 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 13609
I20260809 09:31:50.554416 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:50.554687 9624 external_mini_cluster.cc:1428] Setting key c76d1348e5480c0581ee9f0a4e7b2830
I20260809 09:31:50.555279 13732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:50.561053 13732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb: Bootstrap starting.
I20260809 09:31:50.562592 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:50.555807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:55961 (local address 127.0.0.1:39789)
0809 09:31:50.556239 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:50.556271 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:50.556303 (+ 32us) server_negotiation.cc:407] Connection header received
0809 09:31:50.556850 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:50.556864 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:50.557162 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:50.557463 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:50.557820 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:50.558725 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:50.559594 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:50.560047 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:50.560253 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:50.561052 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:50.561057 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:50.561067 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:50.561453 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:50.561499 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:50.561522 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:50.561657 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:50.561836 (+ 179us) server_negotiation.cc:299] Negotiation successful
0809 09:31:50.561961 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
I20260809 09:31:50.563166 13732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:50.563944 13732 log.cc:824] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:50.565303 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:50.565919 13732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb: No bootstrap required, opened a new log
I20260809 09:31:50.569732 13732 raft_consensus.cc:348] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e783d66c97d04b93b0539515186f00cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39789 } }
I20260809 09:31:50.570000 13732 raft_consensus.cc:374] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:50.570062 13732 raft_consensus.cc:729] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e783d66c97d04b93b0539515186f00cb, State: Initialized, Role: FOLLOWER
I20260809 09:31:50.570503 13732 consensus_queue.cc:260] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e783d66c97d04b93b0539515186f00cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39789 } }
I20260809 09:31:50.570667 13732 raft_consensus.cc:388] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:50.570731 13732 raft_consensus.cc:482] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:50.570818 13732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:50.570958 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:50.567164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:37045 (local address 127.0.0.1:39789)
0809 09:31:50.567286 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:50.567289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:50.567457 (+ 168us) server_negotiation.cc:407] Connection header received
0809 09:31:50.567521 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:50.567525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:50.567567 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:50.567647 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:50.567856 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:50.568609 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:50.569383 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:50.569701 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:50.569812 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:50.570554 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:50.570556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:50.570558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:50.570567 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:50.570586 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:50.570596 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:50.570692 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:50.570817 (+ 125us) server_negotiation.cc:299] Negotiation successful
0809 09:31:50.570848 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 09:31:50.571823 13732 raft_consensus.cc:504] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e783d66c97d04b93b0539515186f00cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39789 } }
I20260809 09:31:50.572192 13732 leader_election.cc:302] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e783d66c97d04b93b0539515186f00cb; no voters:
I20260809 09:31:50.572484 13732 leader_election.cc:288] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:50.572606 13751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:50.572822 13751 raft_consensus.cc:686] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [term 1 LEADER]: Becoming Leader. State: Replica: e783d66c97d04b93b0539515186f00cb, State: Running, Role: LEADER
I20260809 09:31:50.573134 13751 consensus_queue.cc:237] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e783d66c97d04b93b0539515186f00cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39789 } }
I20260809 09:31:50.573518 13732 sys_catalog.cc:564] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:50.574501 13753 sys_catalog.cc:455] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e783d66c97d04b93b0539515186f00cb. Latest consensus state: current_term: 1 leader_uuid: "e783d66c97d04b93b0539515186f00cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e783d66c97d04b93b0539515186f00cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39789 } } }
I20260809 09:31:50.574606 13753 sys_catalog.cc:458] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:50.574961 13765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:50.574965 13752 sys_catalog.cc:455] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e783d66c97d04b93b0539515186f00cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e783d66c97d04b93b0539515186f00cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39789 } } }
I20260809 09:31:50.575605 13752 sys_catalog.cc:458] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:50.577992 13765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:50.582338 13765 catalog_manager.cc:1347] Generated new cluster ID: 56326bc97cfc40788fa2a1e05d29a110
I20260809 09:31:50.582412 13765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:50.591787 13765 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:50.592545 13765 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:50.608139 13765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e783d66c97d04b93b0539515186f00cb: Generated new TSK 0
I20260809 09:31:50.608713 13765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:50.618886 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 13609
W20260809 09:31:50.636006 13590 connection.cc:535] client connection to 127.0.0.1:39789 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39789): Connection reset by peer (error 104)
2026-08-09T09:31:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1528 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:50 dist-test-slave-7vjv krb5kdc[13787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:50 dist-test-slave-7vjv krb5kdc[13787](info): set up 2 sockets
Aug 09 09:31:50 dist-test-slave-7vjv krb5kdc[13787](info): commencing operation
krb5kdc: starting...
W20260809 09:31:51.766317 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:51 dist-test-slave-7vjv krb5kdc[13787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 09:31:51.874935 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:51.973250 13810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:51.973446 13810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:51.973500 13810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:51.977154 13810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:51.977201 13810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:51.977236 13810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:51.977268 13810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:51.977293 13810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:51.981870 13810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:51.982647 13810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:51.983433 13810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:51.990607 13824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:51.990633 13817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:51.990787 13821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:51.990826 13810 server_base.cc:1034] running on GCE node
I20260809 09:31:51.991369 13810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:51.992318 13810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:51.993506 13810 hybrid_clock.cc:648] HybridClock initialized: now 1786267911993475 us; error 49 us; skew 500 ppm
Aug 09 09:31:51 dist-test-slave-7vjv krb5kdc[13787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:51.996292 13810 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 09:31:51.997046 13810 webserver.cc:460] Webserver started at http://127.0.0.1:46775/ using document root <none> and password file <none>
I20260809 09:31:51.997511 13810 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:51.997587 13810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:51.997772 13810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:51.999406 13810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "4c074642bab542c58fe33dc21d9698d7"
format_stamp: "Formatted at 2026-08-09 09:31:51 on dist-test-slave-7vjv"
server_key: "72b238099186d66f6b7c8f004f464c72"
server_key_iv: "da8acba242719aa3bfc7f2252ce0d5ef"
server_key_version: "encryptionkey@0"
I20260809 09:31:51.999900 13810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "4c074642bab542c58fe33dc21d9698d7"
format_stamp: "Formatted at 2026-08-09 09:31:51 on dist-test-slave-7vjv"
server_key: "72b238099186d66f6b7c8f004f464c72"
server_key_iv: "da8acba242719aa3bfc7f2252ce0d5ef"
server_key_version: "encryptionkey@0"
I20260809 09:31:52.003258 13810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260809 09:31:52.005364 13834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:52.006381 13810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:31:52.006515 13810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "4c074642bab542c58fe33dc21d9698d7"
format_stamp: "Formatted at 2026-08-09 09:31:51 on dist-test-slave-7vjv"
server_key: "72b238099186d66f6b7c8f004f464c72"
server_key_iv: "da8acba242719aa3bfc7f2252ce0d5ef"
server_key_version: "encryptionkey@0"
I20260809 09:31:52.006610 13810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:52.010591 13810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:52.013062 13810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:52.013221 13810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:52.019392 13810 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41955
I20260809 09:31:52.019431 13927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41955 every 8 connection(s)
I20260809 09:31:52.020248 13810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:52.022585 13933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:52.027297 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7: Bootstrap starting.
I20260809 09:31:52.029578 13933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:52.029650 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 13810
I20260809 09:31:52.029804 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:52.030144 9624 external_mini_cluster.cc:1428] Setting key 58981223bbacfc454156a52a656c6658
I20260809 09:31:52.030465 13933 log.cc:824] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:52.032655 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7: No bootstrap required, opened a new log
I20260809 09:31:52.034356 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:52.031096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:35485 (local address 127.0.0.1:41955)
0809 09:31:52.031509 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:52.031521 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:52.031552 (+ 31us) server_negotiation.cc:407] Connection header received
0809 09:31:52.032087 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:52.032154 (+ 67us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:52.032382 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:52.032662 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:52.033356 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:52.033360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:52.033369 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:52.033378 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:52.033418 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:52.033434 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:52.033554 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:52.033655 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 09:31:52.033801 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":159,"threads_started":1}
I20260809 09:31:52.036165 13933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c074642bab542c58fe33dc21d9698d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41955 } }
I20260809 09:31:52.036444 13933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:52.036504 13933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c074642bab542c58fe33dc21d9698d7, State: Initialized, Role: FOLLOWER
I20260809 09:31:52.036891 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:52.036895 13933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c074642bab542c58fe33dc21d9698d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41955 } }
I20260809 09:31:52.037058 13933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:52.037117 13933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:52.037199 13933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:52.038156 13933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c074642bab542c58fe33dc21d9698d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41955 } }
I20260809 09:31:52.038529 13933 leader_election.cc:302] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c074642bab542c58fe33dc21d9698d7; no voters:
I20260809 09:31:52.038831 13933 leader_election.cc:288] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:52.038877 13949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:52.039036 13949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [term 1 LEADER]: Becoming Leader. State: Replica: 4c074642bab542c58fe33dc21d9698d7, State: Running, Role: LEADER
I20260809 09:31:52.039290 13949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c074642bab542c58fe33dc21d9698d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41955 } }
I20260809 09:31:52.039964 13933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:52.040405 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c074642bab542c58fe33dc21d9698d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c074642bab542c58fe33dc21d9698d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41955 } } }
I20260809 09:31:52.040580 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:52.040398 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c074642bab542c58fe33dc21d9698d7. Latest consensus state: current_term: 1 leader_uuid: "4c074642bab542c58fe33dc21d9698d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c074642bab542c58fe33dc21d9698d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41955 } } }
I20260809 09:31:52.040923 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:52.042095 13963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:52.042128 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:52.039005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:35025 (local address 127.0.0.1:41955)
0809 09:31:52.039104 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:52.039107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:52.040665 (+ 1558us) server_negotiation.cc:407] Connection header received
0809 09:31:52.040826 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:52.040831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:52.040866 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:52.040913 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:52.041704 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:52.041707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:52.041711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:52.041724 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:52.041746 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:52.041757 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:52.041829 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:52.041986 (+ 157us) server_negotiation.cc:299] Negotiation successful
0809 09:31:52.042021 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 09:31:52.045010 13963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:52.049858 13963 catalog_manager.cc:1347] Generated new cluster ID: 4fbb0617672f47c1ae73e5a3fd4e9582
I20260809 09:31:52.049911 13963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:52.064420 13963 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:52.064689 13963 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:52.070075 13963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c074642bab542c58fe33dc21d9698d7: Generated new TSK 0
I20260809 09:31:52.070673 13963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:52.079644 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 13810
2026-08-09T09:31:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:52 dist-test-slave-7vjv krb5kdc[13984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:52 dist-test-slave-7vjv krb5kdc[13984](info): set up 2 sockets
Aug 09 09:31:52 dist-test-slave-7vjv krb5kdc[13984](info): commencing operation
krb5kdc: starting...
W20260809 09:31:53.254282 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:53 dist-test-slave-7vjv krb5kdc[13984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:31:53.362618 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40431
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:40943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:40431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:53.458725 14007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:53.458910 14007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:53.458943 14007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:53.462287 14007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:53.462328 14007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:53.462344 14007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:53.462358 14007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:53.462376 14007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:53.466579 14007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:40943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:40431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:40431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:53.467384 14007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:53.468190 14007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:53.475379 14020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:53.475409 14007 server_base.cc:1034] running on GCE node
W20260809 09:31:53.475405 14026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:53.475445 14021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:53.476189 14007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:53.477263 14007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:53.478427 14007 hybrid_clock.cc:648] HybridClock initialized: now 1786267913478406 us; error 30 us; skew 500 ppm
Aug 09 09:31:53 dist-test-slave-7vjv krb5kdc[13984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267913, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:53.481004 14007 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:31:53.481803 14007 webserver.cc:460] Webserver started at http://127.9.102.62:37993/ using document root <none> and password file <none>
I20260809 09:31:53.482302 14007 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:53.482390 14007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:53.482596 14007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:53.484439 14007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "6cdd0af648a4486488857adc75f11581"
format_stamp: "Formatted at 2026-08-09 09:31:53 on dist-test-slave-7vjv"
server_key: "d5e5347f2d219ca43218618d1fa8d28e"
server_key_iv: "c762e1fa4ddd0785e25302d078bd785d"
server_key_version: "encryptionkey@0"
I20260809 09:31:53.484988 14007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "6cdd0af648a4486488857adc75f11581"
format_stamp: "Formatted at 2026-08-09 09:31:53 on dist-test-slave-7vjv"
server_key: "d5e5347f2d219ca43218618d1fa8d28e"
server_key_iv: "c762e1fa4ddd0785e25302d078bd785d"
server_key_version: "encryptionkey@0"
I20260809 09:31:53.488476 14007 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:31:53.490742 14039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:53.491858 14007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:53.491991 14007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "6cdd0af648a4486488857adc75f11581"
format_stamp: "Formatted at 2026-08-09 09:31:53 on dist-test-slave-7vjv"
server_key: "d5e5347f2d219ca43218618d1fa8d28e"
server_key_iv: "c762e1fa4ddd0785e25302d078bd785d"
server_key_version: "encryptionkey@0"
I20260809 09:31:53.492094 14007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:53.502111 14007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:53.504622 14007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:53.504809 14007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:53.511102 14007 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:40431
I20260809 09:31:53.511116 14129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:40431 every 8 connection(s)
I20260809 09:31:53.511940 14007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:53.514400 14130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:53.517725 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 14007
I20260809 09:31:53.517880 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:53.518187 9624 external_mini_cluster.cc:1428] Setting key ffcf1e55070bb68e18324ba73582f8a4
I20260809 09:31:53.519850 14130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581: Bootstrap starting.
I20260809 09:31:53.521981 14130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:53.522658 14130 log.cc:824] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:53.524580 14130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581: No bootstrap required, opened a new log
Aug 09 09:31:53 dist-test-slave-7vjv krb5kdc[13984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:53.528812 14130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdd0af648a4486488857adc75f11581" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40431 } }
I20260809 09:31:53.529065 14130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:53.529136 14130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdd0af648a4486488857adc75f11581, State: Initialized, Role: FOLLOWER
I20260809 09:31:53.529541 14130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdd0af648a4486488857adc75f11581" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40431 } }
I20260809 09:31:53.529695 14130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:53.529760 14130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:53.529853 14130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:53.530853 14130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdd0af648a4486488857adc75f11581" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40431 } }
I20260809 09:31:53.531193 14130 leader_election.cc:302] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cdd0af648a4486488857adc75f11581; no voters:
I20260809 09:31:53.531461 14130 leader_election.cc:288] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:53.531596 14137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:53.531788 14137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [term 1 LEADER]: Becoming Leader. State: Replica: 6cdd0af648a4486488857adc75f11581, State: Running, Role: LEADER
I20260809 09:31:53.532058 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:53.519404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60741 (local address 127.9.102.62:40431)
0809 09:31:53.519795 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:53.519810 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:53.519849 (+ 39us) server_negotiation.cc:407] Connection header received
0809 09:31:53.520455 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:53.520471 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:53.520752 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:53.521002 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:53.521773 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:53.522750 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:53.523711 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:53.524203 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:53.524318 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:53.526942 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:53.526948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:53.526963 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:53.526999 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:53.528918 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:53.529531 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:53.529537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:53.529542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:53.529673 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:53.530150 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:53.530155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:53.530156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:53.530620 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:53.530819 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:53.531031 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 09:31:53.531236 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":121,"threads_started":1}
I20260809 09:31:53.532074 14137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdd0af648a4486488857adc75f11581" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40431 } }
I20260809 09:31:53.532531 14130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:53.534742 14140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cdd0af648a4486488857adc75f11581. Latest consensus state: current_term: 1 leader_uuid: "6cdd0af648a4486488857adc75f11581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdd0af648a4486488857adc75f11581" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40431 } } }
I20260809 09:31:53.534997 14140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:53.535048 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:53.534907 14141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cdd0af648a4486488857adc75f11581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdd0af648a4486488857adc75f11581" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 40431 } } }
I20260809 09:31:53.535249 14141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581 [sys.catalog]: This master's current role is: LEADER
W20260809 09:31:53.538038 14160 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:31:53.538125 14160 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:31:53.538179 14161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:53.538954 14161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:53.544238 14161 catalog_manager.cc:1347] Generated new cluster ID: 8274d3d023814d9e9f18011bfe2742e0
I20260809 09:31:53.544299 14161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:53.548202 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:53.536908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48813 (local address 127.9.102.62:40431)
0809 09:31:53.537020 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:53.537023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:53.540344 (+ 3321us) server_negotiation.cc:407] Connection header received
0809 09:31:53.540455 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:53.540457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:53.540494 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:53.540621 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:53.541506 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:53.542511 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:53.543368 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:53.543587 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:53.543713 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:53.545172 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:53.545175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:53.545177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:53.545197 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:53.546548 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:53.547023 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:53.547027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:53.547028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:53.547103 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:53.547394 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:53.547397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:53.547398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:53.547548 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:53.547636 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:53.547961 (+ 325us) server_negotiation.cc:299] Negotiation successful
0809 09:31:53.548051 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 09:31:53.550365 14161 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:53.551287 14161 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:53.558136 14161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cdd0af648a4486488857adc75f11581: Generated new TSK 0
I20260809 09:31:53.558701 14161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:53.561856 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 14007
W20260809 09:31:53.570443 13982 connection.cc:535] client connection to 127.9.102.62:40431 recv error: Network error: failed to read from TLS socket (remote: 127.9.102.62:40431): Connection reset by peer (error 104)
2026-08-09T09:31:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:53 dist-test-slave-7vjv krb5kdc[14183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:53 dist-test-slave-7vjv krb5kdc[14183](info): set up 2 sockets
Aug 09 09:31:53 dist-test-slave-7vjv krb5kdc[14183](info): commencing operation
krb5kdc: starting...
W20260809 09:31:54.689558 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:54 dist-test-slave-7vjv krb5kdc[14183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:31:54.796316 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:37759
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:43313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:37759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:54.896697 14206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:54.896920 14206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:54.896956 14206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:54.900383 14206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:54.900434 14206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:54.900451 14206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:54.900467 14206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:54.900485 14206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:54.905077 14206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:43313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:37759
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:37759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:54.905798 14206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:54.906579 14206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:54.913908 14221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:54.914009 14216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:54.914012 14215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:54.914114 14206 server_base.cc:1034] running on GCE node
I20260809 09:31:54.914662 14206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:54.915551 14206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:54.916746 14206 hybrid_clock.cc:648] HybridClock initialized: now 1786267914916729 us; error 29 us; skew 500 ppm
Aug 09 09:31:54 dist-test-slave-7vjv krb5kdc[14183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267914, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:54.919399 14206 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:31:54.920293 14206 webserver.cc:460] Webserver started at http://127.9.102.62:41099/ using document root <none> and password file <none>
I20260809 09:31:54.920754 14206 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:54.920836 14206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:54.921042 14206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:54.922780 14206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "15b0fe896a36467a9549178def9bb178"
format_stamp: "Formatted at 2026-08-09 09:31:54 on dist-test-slave-7vjv"
server_key: "0b3f2ea591226137bfbd6bc2f5864ac5"
server_key_iv: "fe7956f24cf231c7e3f3e9325903de5e"
server_key_version: "encryptionkey@0"
I20260809 09:31:54.923273 14206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "15b0fe896a36467a9549178def9bb178"
format_stamp: "Formatted at 2026-08-09 09:31:54 on dist-test-slave-7vjv"
server_key: "0b3f2ea591226137bfbd6bc2f5864ac5"
server_key_iv: "fe7956f24cf231c7e3f3e9325903de5e"
server_key_version: "encryptionkey@0"
I20260809 09:31:54.926455 14206 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260809 09:31:54.929001 14236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:54.930462 14206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 09:31:54.930608 14206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "15b0fe896a36467a9549178def9bb178"
format_stamp: "Formatted at 2026-08-09 09:31:54 on dist-test-slave-7vjv"
server_key: "0b3f2ea591226137bfbd6bc2f5864ac5"
server_key_iv: "fe7956f24cf231c7e3f3e9325903de5e"
server_key_version: "encryptionkey@0"
I20260809 09:31:54.930802 14206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:54.960343 14206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:54.962570 14206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:54.962690 14206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:54.968457 14206 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:37759
I20260809 09:31:54.968518 14321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:37759 every 8 connection(s)
I20260809 09:31:54.969221 14206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:54.971302 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 14206
I20260809 09:31:54.971433 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:54.971665 14322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:54.971683 9624 external_mini_cluster.cc:1428] Setting key 2115048fbb084b1d959741e8dfac60ef
I20260809 09:31:54.978138 14322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178: Bootstrap starting.
I20260809 09:31:54.980613 14322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:54.980772 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:54.972896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50361 (local address 127.9.102.62:37759)
0809 09:31:54.973275 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:54.973292 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:54.973329 (+ 37us) server_negotiation.cc:407] Connection header received
0809 09:31:54.974129 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:54.974150 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:54.974522 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:54.974850 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:54.975202 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:54.976507 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:54.977298 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:54.977972 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:54.978143 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:54.978952 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:54.978958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:54.978971 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:54.979422 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:54.979475 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:54.979500 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:54.979640 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:54.979789 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 09:31:54.979903 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":116,"threads_started":1}
I20260809 09:31:54.981511 14322 log.cc:824] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:54.983381 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:54.983417 14322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178: No bootstrap required, opened a new log
I20260809 09:31:54.986793 14322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b0fe896a36467a9549178def9bb178" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37759 } }
I20260809 09:31:54.987008 14322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:54.987087 14322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b0fe896a36467a9549178def9bb178, State: Initialized, Role: FOLLOWER
I20260809 09:31:54.987486 14322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b0fe896a36467a9549178def9bb178" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37759 } }
I20260809 09:31:54.987658 14322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:54.987740 14322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:54.987834 14322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:54.988590 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:54.985185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34281 (local address 127.9.102.62:37759)
0809 09:31:54.985290 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:54.985294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:54.985457 (+ 163us) server_negotiation.cc:407] Connection header received
0809 09:31:54.985577 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:54.985584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:54.985615 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:54.985685 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:54.985917 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:54.986358 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:54.987107 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:54.987416 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:54.987516 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:54.988221 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:54.988224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:54.988226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:54.988237 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:54.988260 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:54.988271 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:54.988352 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:54.988440 (+ 88us) server_negotiation.cc:299] Negotiation successful
0809 09:31:54.988467 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 09:31:54.988783 14322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b0fe896a36467a9549178def9bb178" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37759 } }
I20260809 09:31:54.989073 14322 leader_election.cc:302] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15b0fe896a36467a9549178def9bb178; no voters:
I20260809 09:31:54.989362 14322 leader_election.cc:288] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:54.989426 14340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:54.989621 14340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [term 1 LEADER]: Becoming Leader. State: Replica: 15b0fe896a36467a9549178def9bb178, State: Running, Role: LEADER
I20260809 09:31:54.989925 14340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b0fe896a36467a9549178def9bb178" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37759 } }
I20260809 09:31:54.990442 14322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:54.991255 14343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15b0fe896a36467a9549178def9bb178. Latest consensus state: current_term: 1 leader_uuid: "15b0fe896a36467a9549178def9bb178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b0fe896a36467a9549178def9bb178" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37759 } } }
I20260809 09:31:54.991371 14343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:54.991731 14355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:54.991744 14341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15b0fe896a36467a9549178def9bb178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b0fe896a36467a9549178def9bb178" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 37759 } } }
I20260809 09:31:54.991902 14341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:54.994014 14355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:54.998889 14355 catalog_manager.cc:1347] Generated new cluster ID: 419ca669fa52440ea6f4658a88b31422
I20260809 09:31:54.998962 14355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:55.018924 14355 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:55.020013 14355 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:55.025274 14355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15b0fe896a36467a9549178def9bb178: Generated new TSK 0
I20260809 09:31:55.025873 14355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:55.031437 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 14206
W20260809 09:31:55.049923 14175 connection.cc:535] client connection to 127.9.102.62:37759 recv error: Network error: failed to read from TLS socket (remote: 127.9.102.62:37759): Connection reset by peer (error 104)
2026-08-09T09:31:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1479 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:55 dist-test-slave-7vjv krb5kdc[14372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:55 dist-test-slave-7vjv krb5kdc[14372](info): set up 2 sockets
Aug 09 09:31:55 dist-test-slave-7vjv krb5kdc[14372](info): commencing operation
krb5kdc: starting...
W20260809 09:31:56.189730 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:56 dist-test-slave-7vjv krb5kdc[14372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:31:56.296480 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:45109
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:45109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:56.397292 14397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:56.397504 14397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:56.397543 14397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:56.401062 14397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:56.401101 14397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:56.401117 14397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:56.401131 14397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:56.401161 14397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:56.405692 14397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:45109
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:45109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:56.406450 14397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:56.407259 14397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:56.414601 14405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:56.414597 14409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:56.414608 14407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:56.414796 14397 server_base.cc:1034] running on GCE node
I20260809 09:31:56.415417 14397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:56.416410 14397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:56.417621 14397 hybrid_clock.cc:648] HybridClock initialized: now 1786267916417604 us; error 37 us; skew 500 ppm
Aug 09 09:31:56 dist-test-slave-7vjv krb5kdc[14372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267916, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:56.423758 14397 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:31:56.424584 14397 webserver.cc:460] Webserver started at http://127.9.102.62:37751/ using document root <none> and password file <none>
I20260809 09:31:56.425089 14397 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:56.425205 14397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:56.425448 14397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:56.427296 14397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "a3e86841920449ab833f21aa532969ed"
format_stamp: "Formatted at 2026-08-09 09:31:56 on dist-test-slave-7vjv"
server_key: "1b55b23c21ddadbe33cd3944ca4e5d3e"
server_key_iv: "ad1ec6324f82de602f0e5ef33f3b8cbc"
server_key_version: "encryptionkey@0"
I20260809 09:31:56.427817 14397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "a3e86841920449ab833f21aa532969ed"
format_stamp: "Formatted at 2026-08-09 09:31:56 on dist-test-slave-7vjv"
server_key: "1b55b23c21ddadbe33cd3944ca4e5d3e"
server_key_iv: "ad1ec6324f82de602f0e5ef33f3b8cbc"
server_key_version: "encryptionkey@0"
I20260809 09:31:56.431201 14397 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260809 09:31:56.433445 14418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:56.434502 14397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:31:56.434643 14397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "a3e86841920449ab833f21aa532969ed"
format_stamp: "Formatted at 2026-08-09 09:31:56 on dist-test-slave-7vjv"
server_key: "1b55b23c21ddadbe33cd3944ca4e5d3e"
server_key_iv: "ad1ec6324f82de602f0e5ef33f3b8cbc"
server_key_version: "encryptionkey@0"
I20260809 09:31:56.434753 14397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:56.438843 14397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:56.441316 14397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:56.441483 14397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:56.447382 14397 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:45109
I20260809 09:31:56.447432 14505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:45109 every 8 connection(s)
I20260809 09:31:56.448225 14397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:56.450611 14506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:56.451359 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 14397
I20260809 09:31:56.451491 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:56.451757 9624 external_mini_cluster.cc:1428] Setting key 317f98160bf7879419e7136ee0647714
I20260809 09:31:56.456187 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:56.452927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35251 (local address 127.9.102.62:45109)
0809 09:31:56.453284 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:56.453296 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:56.453325 (+ 29us) server_negotiation.cc:407] Connection header received
0809 09:31:56.453874 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:56.453888 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:56.454122 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:56.454359 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:56.455089 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:56.455093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:56.455104 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:56.455113 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:56.455157 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:56.455173 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:56.455269 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:56.455387 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 09:31:56.455529 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":103,"threads_started":1}
I20260809 09:31:56.456398 14506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed: Bootstrap starting.
I20260809 09:31:56.458657 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:56.458674 14506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:56.459396 14506 log.cc:824] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:56.461310 14506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed: No bootstrap required, opened a new log
I20260809 09:31:56.462332 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:56.460818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35867 (local address 127.9.102.62:45109)
0809 09:31:56.460933 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:56.460939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:56.461130 (+ 191us) server_negotiation.cc:407] Connection header received
0809 09:31:56.461264 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:56.461268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:56.461298 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:56.461383 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:56.461988 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:56.461991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:56.461993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:56.462003 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:56.462032 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:56.462047 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:56.462125 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:56.462193 (+ 68us) server_negotiation.cc:299] Negotiation successful
0809 09:31:56.462226 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 09:31:56.464785 14506 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e86841920449ab833f21aa532969ed" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45109 } }
I20260809 09:31:56.464954 14506 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:56.465046 14506 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e86841920449ab833f21aa532969ed, State: Initialized, Role: FOLLOWER
I20260809 09:31:56.465415 14506 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e86841920449ab833f21aa532969ed" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45109 } }
I20260809 09:31:56.465541 14506 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:56.465612 14506 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:56.465696 14506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:56.466383 14506 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e86841920449ab833f21aa532969ed" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45109 } }
I20260809 09:31:56.466689 14506 leader_election.cc:302] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3e86841920449ab833f21aa532969ed; no voters:
I20260809 09:31:56.466972 14506 leader_election.cc:288] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:56.467119 14527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:56.467339 14527 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [term 1 LEADER]: Becoming Leader. State: Replica: a3e86841920449ab833f21aa532969ed, State: Running, Role: LEADER
I20260809 09:31:56.467602 14527 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e86841920449ab833f21aa532969ed" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45109 } }
I20260809 09:31:56.468071 14506 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:56.469035 14530 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3e86841920449ab833f21aa532969ed. Latest consensus state: current_term: 1 leader_uuid: "a3e86841920449ab833f21aa532969ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e86841920449ab833f21aa532969ed" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45109 } } }
I20260809 09:31:56.469165 14530 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:56.469352 14528 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3e86841920449ab833f21aa532969ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3e86841920449ab833f21aa532969ed" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45109 } } }
I20260809 09:31:56.469677 14528 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:56.469740 14540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:56.472463 14540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:56.477432 14540 catalog_manager.cc:1347] Generated new cluster ID: e4ca513c00f0465f851284617beed573
I20260809 09:31:56.477492 14540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:56.493266 14540 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:56.493569 14540 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:56.497490 14540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3e86841920449ab833f21aa532969ed: Generated new TSK 0
I20260809 09:31:56.498098 14540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:56.504912 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 14397
2026-08-09T09:31:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:56 dist-test-slave-7vjv krb5kdc[14562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:56 dist-test-slave-7vjv krb5kdc[14562](info): set up 2 sockets
Aug 09 09:31:56 dist-test-slave-7vjv krb5kdc[14562](info): commencing operation
krb5kdc: starting...
W20260809 09:31:57.681262 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:57 dist-test-slave-7vjv krb5kdc[14562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:31:57.784991 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:34397
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:34397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:57.881760 14585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:57.881980 14585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:57.882045 14585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:57.885408 14585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:57.885475 14585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:57.885507 14585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:57.885558 14585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:57.885588 14585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:57.890239 14585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:34397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:34397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:57.891127 14585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:57.891984 14585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:57.899385 14599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:57.899380 14594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:57.899420 14585 server_base.cc:1034] running on GCE node
W20260809 09:31:57.899365 14593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:57.900245 14585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:57.901081 14585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:57.902231 14585 hybrid_clock.cc:648] HybridClock initialized: now 1786267917902217 us; error 25 us; skew 500 ppm
Aug 09 09:31:57 dist-test-slave-7vjv krb5kdc[14562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267917, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:57.904626 14585 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:31:57.905383 14585 webserver.cc:460] Webserver started at http://127.9.102.62:40957/ using document root <none> and password file <none>
I20260809 09:31:57.905855 14585 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:57.905910 14585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:57.906111 14585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:57.907941 14585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "819be1e636aa469d90ce4da26b01662a"
format_stamp: "Formatted at 2026-08-09 09:31:57 on dist-test-slave-7vjv"
server_key: "e864be1c7d0c7424c9ac9394a5829508"
server_key_iv: "9b2283d6450846539534cad2e6e75d1f"
server_key_version: "encryptionkey@0"
I20260809 09:31:57.908474 14585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "819be1e636aa469d90ce4da26b01662a"
format_stamp: "Formatted at 2026-08-09 09:31:57 on dist-test-slave-7vjv"
server_key: "e864be1c7d0c7424c9ac9394a5829508"
server_key_iv: "9b2283d6450846539534cad2e6e75d1f"
server_key_version: "encryptionkey@0"
I20260809 09:31:57.911592 14585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:31:57.913542 14611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:57.914505 14585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 09:31:57.914633 14585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "819be1e636aa469d90ce4da26b01662a"
format_stamp: "Formatted at 2026-08-09 09:31:57 on dist-test-slave-7vjv"
server_key: "e864be1c7d0c7424c9ac9394a5829508"
server_key_iv: "9b2283d6450846539534cad2e6e75d1f"
server_key_version: "encryptionkey@0"
I20260809 09:31:57.914723 14585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:57.940902 14585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:57.943116 14585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:57.943281 14585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:57.949386 14585 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:34397
I20260809 09:31:57.949406 14697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:34397 every 8 connection(s)
I20260809 09:31:57.950133 14585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:57.952507 14698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:57.957031 14698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a: Bootstrap starting.
I20260809 09:31:57.959003 14698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:57.959637 14698 log.cc:824] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:57.960018 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 14585
I20260809 09:31:57.960196 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:57.960476 9624 external_mini_cluster.cc:1428] Setting key c24e943657265e0ee386b9be8fa8bf22
I20260809 09:31:57.961547 14698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a: No bootstrap required, opened a new log
I20260809 09:31:57.964934 14698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819be1e636aa469d90ce4da26b01662a" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 34397 } }
I20260809 09:31:57.965150 14698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:57.965199 14698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819be1e636aa469d90ce4da26b01662a, State: Initialized, Role: FOLLOWER
I20260809 09:31:57.965579 14698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819be1e636aa469d90ce4da26b01662a" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 34397 } }
I20260809 09:31:57.965689 14698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:57.965777 14698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:57.965873 14698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:57.966867 14698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819be1e636aa469d90ce4da26b01662a" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 34397 } }
I20260809 09:31:57.967253 14698 leader_election.cc:302] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 819be1e636aa469d90ce4da26b01662a; no voters:
I20260809 09:31:57.967585 14698 leader_election.cc:288] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:57.967660 14706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:57.967969 14706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [term 1 LEADER]: Becoming Leader. State: Replica: 819be1e636aa469d90ce4da26b01662a, State: Running, Role: LEADER
Aug 09 09:31:57 dist-test-slave-7vjv krb5kdc[14562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:31:57.968379 14706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819be1e636aa469d90ce4da26b01662a" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 34397 } }
I20260809 09:31:57.968875 14698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:57.971217 14709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "819be1e636aa469d90ce4da26b01662a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819be1e636aa469d90ce4da26b01662a" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 34397 } } }
I20260809 09:31:57.971346 14709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:57.971926 14710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 819be1e636aa469d90ce4da26b01662a. Latest consensus state: current_term: 1 leader_uuid: "819be1e636aa469d90ce4da26b01662a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819be1e636aa469d90ce4da26b01662a" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 34397 } } }
I20260809 09:31:57.972018 14710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:57.973449 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:57.961377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44217 (local address 127.9.102.62:34397)
0809 09:31:57.961742 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:57.961757 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:57.961784 (+ 27us) server_negotiation.cc:407] Connection header received
0809 09:31:57.962274 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:57.962287 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:57.962541 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:57.962761 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:57.963568 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:57.964404 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:57.965452 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:57.965973 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:57.966092 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:57.968891 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:57.968896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:57.968909 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:57.968935 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:57.970700 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:57.971223 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:57.971230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:57.971237 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:57.971372 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:57.971706 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:57.971709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:57.971711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:57.972194 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:57.972380 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:57.972661 (+ 281us) server_negotiation.cc:299] Negotiation successful
0809 09:31:57.972893 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":128,"threads_started":1}
W20260809 09:31:57.973507 14731 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:31:57.973579 14731 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:31:57.973640 14732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:57.974262 14732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:57.976002 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:57.979382 14732 catalog_manager.cc:1347] Generated new cluster ID: b185347a277e41b9a9a8ec2c87132dbb
I20260809 09:31:57.979437 14732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:57.984790 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:57.978296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38351 (local address 127.9.102.62:34397)
0809 09:31:57.978391 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:57.978395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:57.978592 (+ 197us) server_negotiation.cc:407] Connection header received
0809 09:31:57.978685 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:57.978688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:57.978728 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:57.978811 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:57.979554 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:57.979993 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:57.980769 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:57.980969 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:57.981063 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:57.982194 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:57.982197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:57.982199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:31:57.982223 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:57.983381 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:57.983734 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:57.983738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:57.983739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:57.983811 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:31:57.984042 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:31:57.984047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:31:57.984051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:31:57.984222 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:57.984336 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:57.984542 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 09:31:57.984639 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 09:31:58.001052 14732 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:58.001837 14732 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:58.005744 14732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 819be1e636aa469d90ce4da26b01662a: Generated new TSK 0
I20260809 09:31:58.006263 14732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:58.009370 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 14585
2026-08-09T09:31:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1506 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:58 dist-test-slave-7vjv krb5kdc[14755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:58 dist-test-slave-7vjv krb5kdc[14755](info): set up 2 sockets
Aug 09 09:31:58 dist-test-slave-7vjv krb5kdc[14755](info): commencing operation
krb5kdc: starting...
W20260809 09:31:59.179850 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:31:59 dist-test-slave-7vjv krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:31:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:31:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:31:59.287559 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:45371
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:38757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:45371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:31:59.388237 14775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:31:59.388448 14775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:31:59.388482 14775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:31:59.391956 14775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:31:59.391999 14775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:31:59.392014 14775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:31:59.392050 14775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:31:59.392063 14775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:31:59.396761 14775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:38757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:45371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:45371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:31:59.397583 14775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:31:59.398392 14775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:31:59.406265 14787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:31:59.406299 14789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:59.406481 14775 server_base.cc:1034] running on GCE node
W20260809 09:31:59.406301 14786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:31:59.407104 14775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:31:59.408020 14775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:31:59.409168 14775 hybrid_clock.cc:648] HybridClock initialized: now 1786267919409144 us; error 36 us; skew 500 ppm
Aug 09 09:31:59 dist-test-slave-7vjv krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267919, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:31:59.411657 14775 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:31:59.412483 14775 webserver.cc:460] Webserver started at http://127.9.102.62:44729/ using document root <none> and password file <none>
I20260809 09:31:59.412914 14775 fs_manager.cc:359] Metadata directory not provided
I20260809 09:31:59.412963 14775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:31:59.413097 14775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:31:59.414731 14775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "179fce8ce99b49cda7c1a5a033208433"
format_stamp: "Formatted at 2026-08-09 09:31:59 on dist-test-slave-7vjv"
server_key: "76eec1f906a9236bbbc805b29aab03ec"
server_key_iv: "6ff82f4f1f0f0d92efc8ba3cf21264c4"
server_key_version: "encryptionkey@0"
I20260809 09:31:59.415211 14775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "179fce8ce99b49cda7c1a5a033208433"
format_stamp: "Formatted at 2026-08-09 09:31:59 on dist-test-slave-7vjv"
server_key: "76eec1f906a9236bbbc805b29aab03ec"
server_key_iv: "6ff82f4f1f0f0d92efc8ba3cf21264c4"
server_key_version: "encryptionkey@0"
I20260809 09:31:59.418404 14775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 09:31:59.420451 14799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:31:59.421515 14775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:31:59.421664 14775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "179fce8ce99b49cda7c1a5a033208433"
format_stamp: "Formatted at 2026-08-09 09:31:59 on dist-test-slave-7vjv"
server_key: "76eec1f906a9236bbbc805b29aab03ec"
server_key_iv: "6ff82f4f1f0f0d92efc8ba3cf21264c4"
server_key_version: "encryptionkey@0"
I20260809 09:31:59.421758 14775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:31:59.438028 14775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:31:59.440834 14775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:31:59.441033 14775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:31:59.447582 14775 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:45371
I20260809 09:31:59.447607 14883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:45371 every 8 connection(s)
I20260809 09:31:59.448426 14775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:31:59.450774 14885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:31:59.452566 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 14775
I20260809 09:31:59.452726 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:31:59.452996 9624 external_mini_cluster.cc:1428] Setting key 5cc4ebd32c83094191e22f98b08129c6
I20260809 09:31:59.456174 14885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433: Bootstrap starting.
I20260809 09:31:59.458194 14885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433: Neither blocks nor log segments found. Creating new log.
I20260809 09:31:59.458884 14885 log.cc:824] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433: Log is configured to *not* fsync() on all Append() calls
I20260809 09:31:59.460700 14885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433: No bootstrap required, opened a new log
I20260809 09:31:59.461524 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:59.454180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37321 (local address 127.9.102.62:45371)
0809 09:31:59.454575 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:59.454591 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:59.454621 (+ 30us) server_negotiation.cc:407] Connection header received
0809 09:31:59.455393 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:59.455411 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:59.455709 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:59.455994 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:59.456393 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:59.457263 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:59.458138 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:59.458640 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:59.458793 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:59.459961 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:59.459966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:59.459978 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:59.460388 (+ 410us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:59.460436 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:59.460458 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:59.460557 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:59.460713 (+ 156us) server_negotiation.cc:299] Negotiation successful
0809 09:31:59.460925 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
I20260809 09:31:59.463866 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:31:59.463903 14885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179fce8ce99b49cda7c1a5a033208433" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45371 } }
I20260809 09:31:59.464140 14885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:31:59.464197 14885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179fce8ce99b49cda7c1a5a033208433, State: Initialized, Role: FOLLOWER
I20260809 09:31:59.464617 14885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179fce8ce99b49cda7c1a5a033208433" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45371 } }
I20260809 09:31:59.464776 14885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:31:59.464841 14885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:31:59.464941 14885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:31:59.465905 14885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179fce8ce99b49cda7c1a5a033208433" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45371 } }
I20260809 09:31:59.466265 14885 leader_election.cc:302] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 179fce8ce99b49cda7c1a5a033208433; no voters:
I20260809 09:31:59.466571 14885 leader_election.cc:288] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:31:59.466626 14904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:31:59.466818 14904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [term 1 LEADER]: Becoming Leader. State: Replica: 179fce8ce99b49cda7c1a5a033208433, State: Running, Role: LEADER
I20260809 09:31:59.467176 14904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179fce8ce99b49cda7c1a5a033208433" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45371 } }
I20260809 09:31:59.467957 14885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:31:59.468978 14905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 179fce8ce99b49cda7c1a5a033208433. Latest consensus state: current_term: 1 leader_uuid: "179fce8ce99b49cda7c1a5a033208433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179fce8ce99b49cda7c1a5a033208433" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45371 } } }
I20260809 09:31:59.469396 14905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:59.469280 14907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "179fce8ce99b49cda7c1a5a033208433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179fce8ce99b49cda7c1a5a033208433" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 45371 } } }
I20260809 09:31:59.469714 14907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433 [sys.catalog]: This master's current role is: LEADER
I20260809 09:31:59.469805 14916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:31:59.471158 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:31:59.465949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40617 (local address 127.9.102.62:45371)
0809 09:31:59.466093 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 09:31:59.466097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:31:59.466202 (+ 105us) server_negotiation.cc:407] Connection header received
0809 09:31:59.466296 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:31:59.466298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:31:59.466338 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:31:59.466432 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:31:59.466809 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:59.467447 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:59.468320 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:31:59.468848 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:31:59.468985 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:31:59.470658 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:31:59.470662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:31:59.470665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:31:59.470679 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:31:59.470707 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:31:59.470721 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:31:59.470834 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 09:31:59.470965 (+ 131us) server_negotiation.cc:299] Negotiation successful
0809 09:31:59.471016 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 09:31:59.473057 14916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:31:59.478228 14916 catalog_manager.cc:1347] Generated new cluster ID: 1303e0b3325447638606d7393179b2c0
I20260809 09:31:59.478296 14916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:31:59.485745 14916 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:31:59.486488 14916 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:31:59.492127 14916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 179fce8ce99b49cda7c1a5a033208433: Generated new TSK 0
I20260809 09:31:59.492700 14916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:31:59.496402 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 14775
2026-08-09T09:31:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:31:59 dist-test-slave-7vjv krb5kdc[14943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:31:59 dist-test-slave-7vjv krb5kdc[14943](info): set up 2 sockets
Aug 09 09:31:59 dist-test-slave-7vjv krb5kdc[14943](info): commencing operation
krb5kdc: starting...
W20260809 09:32:00.727047 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:32:00 dist-test-slave-7vjv krb5kdc[14943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:32:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:32:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:32:00.833151 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:42557
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:42557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:32:00.929705 14960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:32:00.929893 14960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:32:00.929925 14960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:32:00.933439 14960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:32:00.933487 14960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:32:00.933506 14960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:32:00.933521 14960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:32:00.933538 14960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:32:00.937947 14960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:42557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:42557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:32:00.938704 14960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:32:00.939465 14960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:32:00.946693 14970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:32:00.946715 14967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:32:00.946713 14972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:32:00.946800 14960 server_base.cc:1034] running on GCE node
I20260809 09:32:00.947568 14960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:32:00.948688 14960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:32:00.949849 14960 hybrid_clock.cc:648] HybridClock initialized: now 1786267920949831 us; error 31 us; skew 500 ppm
Aug 09 09:32:00 dist-test-slave-7vjv krb5kdc[14943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267920, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:32:00.952745 14960 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:32:00.953482 14960 webserver.cc:460] Webserver started at http://127.9.102.62:40573/ using document root <none> and password file <none>
I20260809 09:32:00.953951 14960 fs_manager.cc:359] Metadata directory not provided
I20260809 09:32:00.954008 14960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:32:00.954228 14960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:32:00.955987 14960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "cc5ab26715d34cb382d039bf6481b018"
format_stamp: "Formatted at 2026-08-09 09:32:00 on dist-test-slave-7vjv"
server_key: "30d777d2204e685021e6a51248626c11"
server_key_iv: "fa87cceb7d15e11b1c648dac483947bc"
server_key_version: "encryptionkey@0"
I20260809 09:32:00.956604 14960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "cc5ab26715d34cb382d039bf6481b018"
format_stamp: "Formatted at 2026-08-09 09:32:00 on dist-test-slave-7vjv"
server_key: "30d777d2204e685021e6a51248626c11"
server_key_iv: "fa87cceb7d15e11b1c648dac483947bc"
server_key_version: "encryptionkey@0"
I20260809 09:32:00.959820 14960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:32:00.961874 14986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:32:00.962822 14960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:32:00.962951 14960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "cc5ab26715d34cb382d039bf6481b018"
format_stamp: "Formatted at 2026-08-09 09:32:00 on dist-test-slave-7vjv"
server_key: "30d777d2204e685021e6a51248626c11"
server_key_iv: "fa87cceb7d15e11b1c648dac483947bc"
server_key_version: "encryptionkey@0"
I20260809 09:32:00.963033 14960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:32:00.966956 14960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:32:00.969434 14960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:32:00.969614 14960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:32:00.975822 14960 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:42557
I20260809 09:32:00.975837 15074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:42557 every 8 connection(s)
I20260809 09:32:00.976668 14960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:32:00.977784 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 14960
I20260809 09:32:00.977919 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:32:00.978171 9624 external_mini_cluster.cc:1428] Setting key 1afd5df80a64427a0bcc8f386248463b
I20260809 09:32:00.979351 15076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:32:00.982960 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:00.979364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58391 (local address 127.9.102.62:42557)
0809 09:32:00.979781 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:00.979797 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:00.979833 (+ 36us) server_negotiation.cc:407] Connection header received
0809 09:32:00.980522 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:00.980551 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:00.980859 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:00.981159 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:00.981773 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:00.981778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:00.981791 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:32:00.981802 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:00.981851 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:32:00.981873 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:00.982000 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:00.982063 (+ 63us) server_negotiation.cc:299] Negotiation successful
0809 09:32:00.982189 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260809 09:32:00.985275 15076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018: Bootstrap starting.
I20260809 09:32:00.985536 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:32:00.987457 15076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018: Neither blocks nor log segments found. Creating new log.
I20260809 09:32:00.988184 15076 log.cc:824] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018: Log is configured to *not* fsync() on all Append() calls
I20260809 09:32:00.988940 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:00.987446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53565 (local address 127.9.102.62:42557)
0809 09:32:00.987563 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:00.987568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:00.987756 (+ 188us) server_negotiation.cc:407] Connection header received
0809 09:32:00.987860 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:00.987864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:00.987896 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:00.987969 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:00.988591 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:00.988605 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:00.988607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 09:32:00.988617 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:00.988642 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 09:32:00.988651 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:00.988725 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:00.988805 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 09:32:00.988838 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 09:32:00.990057 15076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018: No bootstrap required, opened a new log
I20260809 09:32:00.993095 15076 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5ab26715d34cb382d039bf6481b018" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42557 } }
I20260809 09:32:00.993283 15076 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 09:32:00.993371 15076 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5ab26715d34cb382d039bf6481b018, State: Initialized, Role: FOLLOWER
I20260809 09:32:00.993798 15076 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5ab26715d34cb382d039bf6481b018" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42557 } }
I20260809 09:32:00.993927 15076 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 09:32:00.993995 15076 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 09:32:00.994084 15076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:32:00.994750 15076 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5ab26715d34cb382d039bf6481b018" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42557 } }
I20260809 09:32:00.995090 15076 leader_election.cc:302] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc5ab26715d34cb382d039bf6481b018; no voters:
I20260809 09:32:00.995365 15076 leader_election.cc:288] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 09:32:00.995476 15088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:32:00.995779 15088 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [term 1 LEADER]: Becoming Leader. State: Replica: cc5ab26715d34cb382d039bf6481b018, State: Running, Role: LEADER
I20260809 09:32:00.996117 15088 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5ab26715d34cb382d039bf6481b018" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42557 } }
I20260809 09:32:00.996483 15076 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:32:00.997153 15088 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc5ab26715d34cb382d039bf6481b018. Latest consensus state: current_term: 1 leader_uuid: "cc5ab26715d34cb382d039bf6481b018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5ab26715d34cb382d039bf6481b018" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42557 } } }
I20260809 09:32:00.997335 15088 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [sys.catalog]: This master's current role is: LEADER
I20260809 09:32:00.997567 15091 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc5ab26715d34cb382d039bf6481b018" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5ab26715d34cb382d039bf6481b018" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 42557 } } }
I20260809 09:32:00.997702 15091 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018 [sys.catalog]: This master's current role is: LEADER
I20260809 09:32:00.998073 15110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:32:01.000655 15110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:32:01.005219 15110 catalog_manager.cc:1347] Generated new cluster ID: 26c7b7597f084bc6913737380a872cb0
I20260809 09:32:01.005271 15110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:32:01.014125 15110 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:32:01.014406 15110 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:32:01.018635 15110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc5ab26715d34cb382d039bf6481b018: Generated new TSK 0
I20260809 09:32:01.019204 15110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 09:32:01.025707 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 14960
2026-08-09T09:32:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1536 ms)
[----------] 18 tests from AuthTokenIssuingTest (26882 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:32:01 dist-test-slave-7vjv krb5kdc[15125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:32:01 dist-test-slave-7vjv krb5kdc[15125](info): set up 2 sockets
Aug 09 09:32:01 dist-test-slave-7vjv krb5kdc[15125](info): commencing operation
krb5kdc: starting...
W20260809 09:32:02.211345 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:32:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:32:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:32:02.318786 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39877
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39877,127.9.102.61:45173,127.9.102.60:37431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:32:02.421847 15150 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 09:32:02.422114 15150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:32:02.422196 15150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:32:02.422278 15150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:32:02.425781 15150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:32:02.425858 15150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:32:02.425896 15150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:32:02.425951 15150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:32:02.425985 15150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:32:02.430502 15150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39877,127.9.102.61:45173,127.9.102.60:37431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:32:02.431375 15150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:32:02.432281 15150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:32:02.439944 15158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:32:02.439942 15157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:32:02.439987 15165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:32:02.440282 15150 server_base.cc:1034] running on GCE node
I20260809 09:32:02.440809 15150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:32:02.441761 15150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:32:02.443053 15150 hybrid_clock.cc:648] HybridClock initialized: now 1786267922443039 us; error 37 us; skew 500 ppm
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:32:02.448438 15150 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:32:02.449126 15150 webserver.cc:460] Webserver started at http://127.9.102.62:34147/ using document root <none> and password file <none>
I20260809 09:32:02.449560 15150 fs_manager.cc:359] Metadata directory not provided
I20260809 09:32:02.449608 15150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:32:02.449740 15150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:32:02.451411 15150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "89541ebe94b64f38ab668de9d57c77c9"
format_stamp: "Formatted at 2026-08-09 09:32:02 on dist-test-slave-7vjv"
server_key: "79a88013fe06e5c02b2405852810eaf3"
server_key_iv: "48b9b8f529bdcc35dfafbb399dc22237"
server_key_version: "encryptionkey@0"
I20260809 09:32:02.451891 15150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "89541ebe94b64f38ab668de9d57c77c9"
format_stamp: "Formatted at 2026-08-09 09:32:02 on dist-test-slave-7vjv"
server_key: "79a88013fe06e5c02b2405852810eaf3"
server_key_iv: "48b9b8f529bdcc35dfafbb399dc22237"
server_key_version: "encryptionkey@0"
I20260809 09:32:02.455156 15150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260809 09:32:02.457219 15182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:32:02.458305 15150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 09:32:02.458426 15150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "89541ebe94b64f38ab668de9d57c77c9"
format_stamp: "Formatted at 2026-08-09 09:32:02 on dist-test-slave-7vjv"
server_key: "79a88013fe06e5c02b2405852810eaf3"
server_key_iv: "48b9b8f529bdcc35dfafbb399dc22237"
server_key_version: "encryptionkey@0"
I20260809 09:32:02.458518 15150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:32:02.472465 15150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:32:02.474818 15150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:32:02.474992 15150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:32:02.481237 15150 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:39877
I20260809 09:32:02.481484 15281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:39877 every 8 connection(s)
I20260809 09:32:02.482079 15150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:32:02.484295 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 15150
I20260809 09:32:02.484439 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:32:02.484599 15283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:32:02.484724 9624 external_mini_cluster.cc:1428] Setting key 5382aa39d42ccfea010e2faf023ac0d9
I20260809 09:32:02.488845 15283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.102.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.61@KRBTEST.COM" created.
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:32:02.502621 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.491951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59678 (local address 127.9.102.62:39877)
0809 09:32:02.492255 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.492265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.492306 (+ 41us) server_negotiation.cc:407] Connection header received
0809 09:32:02.492630 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.492634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.492786 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.492952 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.493716 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.494697 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.495618 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.495947 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.496126 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.498325 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.498334 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.498349 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.498377 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.499889 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.500490 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.500494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.500495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.500604 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.500980 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.500982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.500983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.501282 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:02.501377 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:02.501822 (+ 445us) server_negotiation.cc:299] Negotiation successful
0809 09:32:02.501995 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
I20260809 09:32:02.502624 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.491023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39877 (local address 127.0.0.1:59678)
0809 09:32:02.491500 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.491540 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:02.492363 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:02.492951 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:02.492955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:02.493339 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:02.493584 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.493598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.494833 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.494836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.495504 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.495512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.496121 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.496126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.496275 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.496803 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:02.496817 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:02.498176 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:02.499997 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.500007 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.500016 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.500372 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.500703 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.500709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.500713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.500881 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.501399 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:02.501404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:02.501565 (+ 161us) client_negotiation.cc:769] Sending connection context
0809 09:32:02.501728 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 09:32:02.502102 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":70,"thread_start_us":145,"threads_started":1}
I20260809 09:32:02.504817 15283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:32:02.505451 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.505198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:45173 (local address 127.0.0.1:50366)
0809 09:32:02.505309 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.505402 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 09:32:02.505714 15193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.102.61:45173: connect: Connection refused (error 111)
I20260809 09:32:02.506378 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.506215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:45173 (local address 127.0.0.1:50376)
0809 09:32:02.506296 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.506335 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260809 09:32:02.506583 15283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.61:45173: Network error: Client connection negotiation failed: client connection to 127.9.102.61:45173: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.61.keytab.
Entry for principal kudu/127.9.102.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.61.keytab.
I20260809 09:32:02.510859 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.61:45173
--webserver_interface=127.9.102.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.61.keytab
--principal=kudu/127.9.102.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39877,127.9.102.61:45173,127.9.102.60:37431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:32:02.555930 15283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } attempt: 1
I20260809 09:32:02.556582 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.556366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:45173 (local address 127.0.0.1:50384)
0809 09:32:02.556479 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.556528 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 09:32:02.557408 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.557174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:45173 (local address 127.0.0.1:50400)
0809 09:32:02.557307 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.557359 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260809 09:32:02.557559 15283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.61:45173: Network error: Client connection negotiation failed: client connection to 127.9.102.61:45173: connect: Connection refused (error 111)
W20260809 09:32:02.609921 15293 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 09:32:02.610173 15293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:32:02.610265 15293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:32:02.610340 15293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:32:02.614002 15293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:32:02.614076 15293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:32:02.614120 15293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:32:02.614176 15293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:32:02.614212 15293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:32:02.618765 15293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39877,127.9.102.61:45173,127.9.102.60:37431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.61.keytab
--principal=kudu/127.9.102.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.61:45173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.102.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:32:02.619568 15293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:32:02.620478 15293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 09:32:02.625715 15283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } attempt: 2
I20260809 09:32:02.626379 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.626096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:45173 (local address 127.0.0.1:50408)
0809 09:32:02.626233 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.626298 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 09:32:02.627169 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.626953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:45173 (local address 127.0.0.1:50424)
0809 09:32:02.627055 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.627103 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260809 09:32:02.627343 15283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.61:45173: Network error: Client connection negotiation failed: client connection to 127.9.102.61:45173: connect: Connection refused (error 111)
W20260809 09:32:02.627445 15300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:32:02.627681 15305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:32:02.627460 15299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:32:02.627806 15293 server_base.cc:1034] running on GCE node
I20260809 09:32:02.628367 15293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:32:02.629330 15293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:32:02.630518 15293 hybrid_clock.cc:648] HybridClock initialized: now 1786267922630502 us; error 28 us; skew 500 ppm
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:32:02.632970 15293 init.cc:377] Logged in from keytab as kudu/127.9.102.61@KRBTEST.COM (short username kudu)
I20260809 09:32:02.633706 15293 webserver.cc:460] Webserver started at http://127.9.102.61:33153/ using document root <none> and password file <none>
I20260809 09:32:02.634205 15293 fs_manager.cc:359] Metadata directory not provided
I20260809 09:32:02.634292 15293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:32:02.634501 15293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:32:02.636438 15293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/data/instance:
uuid: "215caf8187bc44adae3c7cbc33d8e27c"
format_stamp: "Formatted at 2026-08-09 09:32:02 on dist-test-slave-7vjv"
server_key: "9586bc4fb9cd6f170964b5e356c0e570"
server_key_iv: "edde995e70c05a1c2efb1a616935ec8e"
server_key_version: "encryptionkey@0"
I20260809 09:32:02.636996 15293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/wal/instance:
uuid: "215caf8187bc44adae3c7cbc33d8e27c"
format_stamp: "Formatted at 2026-08-09 09:32:02 on dist-test-slave-7vjv"
server_key: "9586bc4fb9cd6f170964b5e356c0e570"
server_key_iv: "edde995e70c05a1c2efb1a616935ec8e"
server_key_version: "encryptionkey@0"
I20260809 09:32:02.640142 15293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 09:32:02.642108 15317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:32:02.643133 15293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:32:02.643262 15293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/wal
uuid: "215caf8187bc44adae3c7cbc33d8e27c"
format_stamp: "Formatted at 2026-08-09 09:32:02 on dist-test-slave-7vjv"
server_key: "9586bc4fb9cd6f170964b5e356c0e570"
server_key_iv: "edde995e70c05a1c2efb1a616935ec8e"
server_key_version: "encryptionkey@0"
I20260809 09:32:02.643358 15293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:32:02.662413 15293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:32:02.664637 15293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:32:02.664806 15293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:32:02.670974 15293 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.61:45173
I20260809 09:32:02.671216 15410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.61:45173 every 8 connection(s)
I20260809 09:32:02.671788 15293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
I20260809 09:32:02.674134 15414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:32:02.676064 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 15293
I20260809 09:32:02.676227 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-1/wal/instance
I20260809 09:32:02.676477 9624 external_mini_cluster.cc:1428] Setting key bfac966593e7453d234e9fc97ceacf5a
I20260809 09:32:02.677309 15414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.102.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.60@KRBTEST.COM" created.
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.61@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:32:02.691504 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.678982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.9.102.62:39877)
0809 09:32:02.679111 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.679116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.679843 (+ 727us) server_negotiation.cc:407] Connection header received
0809 09:32:02.680652 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.680655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.680705 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.680795 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.681705 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.682159 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.683104 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.683313 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.683406 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.687745 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.687748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.687750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.687774 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.689504 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.689992 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.690015 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.690018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.690122 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.690461 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.690465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.690467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.690647 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:02.690756 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:02.691242 (+ 486us) server_negotiation.cc:299] Negotiation successful
0809 09:32:02.691360 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:32:02.692202 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.679291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39877 (local address 127.0.0.1:59684)
0809 09:32:02.679721 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.679755 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:02.680477 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:02.680935 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:02.680940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:02.681206 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:02.681581 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.681592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.682279 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.682282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.682984 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.682992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.683452 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.683456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.683593 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.684141 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:02.684164 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:02.687580 (+ 3416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:02.689631 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.689636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.689647 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.689871 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.690218 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.690222 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.690224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.690362 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.690755 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:02.690762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:02.690987 (+ 225us) client_negotiation.cc:769] Sending connection context
0809 09:32:02.691170 (+ 183us) client_negotiation.cc:240] Negotiation successful
0809 09:32:02.691457 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":136,"threads_started":1}
I20260809 09:32:02.693607 15414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.61@KRBTEST.COM for kudu/127.9.102.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.60.keytab.
Entry for principal kudu/127.9.102.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.60.keytab.
I20260809 09:32:02.701627 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.60:37431
--webserver_interface=127.9.102.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.60.keytab
--principal=kudu/127.9.102.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:39877,127.9.102.61:45173,127.9.102.60:37431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:32:02.702134 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.694021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:45173 (local address 127.0.0.1:50434)
0809 09:32:02.694146 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.694156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:02.694206 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:02.694659 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:02.694661 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:02.694822 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:02.694878 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.694883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.695778 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.695781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.696400 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.696409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.696951 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.696954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.697037 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.697396 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:02.697406 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:02.698584 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:02.700455 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.700457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.700458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.700700 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.701064 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.701067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.701068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.701139 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.701638 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:02.701641 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:02.701728 (+ 87us) client_negotiation.cc:769] Sending connection context
0809 09:32:02.701854 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 09:32:02.701974 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260809 09:32:02.702255 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.694117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.9.102.61:45173)
0809 09:32:02.694383 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.694390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.694405 (+ 15us) server_negotiation.cc:407] Connection header received
0809 09:32:02.694433 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.694436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.694554 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.694676 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.694979 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.695637 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.696506 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.696847 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.696976 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.698704 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.698712 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.698739 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.698759 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.700356 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.700852 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.700860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.700864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.700957 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.701246 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.701251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.701252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.701460 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:02.701669 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:02.701967 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 09:32:02.702083 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":93,"threads_started":1}
I20260809 09:32:02.703723 15414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:32:02.704317 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.704128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55110)
0809 09:32:02.704205 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.704271 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260809 09:32:02.704495 15320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
I20260809 09:32:02.705231 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.705029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55114)
0809 09:32:02.705122 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.705186 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 09:32:02.705381 15414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:37431: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
I20260809 09:32:02.718497 15283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } attempt: 3
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.61@KRBTEST.COM
I20260809 09:32:02.726204 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.718973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.9.102.61:45173)
0809 09:32:02.719085 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.719089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.719101 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:32:02.719214 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.719216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.719251 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.719331 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.719701 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.720230 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.721309 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.721551 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.721650 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.723429 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.723432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.723436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.723459 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.724770 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.725168 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.725177 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.725179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.725241 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.725495 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.725499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.725501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.725645 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:02.725741 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:02.725975 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 09:32:02.726077 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 09:32:02.726205 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.718917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:45173 (local address 127.0.0.1:50444)
0809 09:32:02.719027 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.719048 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:02.719121 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:02.719349 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:02.719351 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:02.719499 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:02.719595 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.719600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.720343 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.720346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.721192 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.721202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.721653 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.721656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.721755 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.722148 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:02.722158 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:02.723296 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:02.724872 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.724876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.724878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.725072 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.725326 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.725329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.725331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.725411 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.725736 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:02.725739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:02.725829 (+ 90us) client_negotiation.cc:769] Sending connection context
0809 09:32:02.725929 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 09:32:02.726055 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 09:32:02.727355 15283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:32:02.727912 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.727722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55126)
0809 09:32:02.727813 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.727856 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260809 09:32:02.728521 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.728304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55138)
0809 09:32:02.728428 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.728470 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260809 09:32:02.728657 15283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:37431: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
I20260809 09:32:02.754637 15414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } attempt: 1
I20260809 09:32:02.755308 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.755057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55148)
0809 09:32:02.755172 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.755226 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 09:32:02.755973 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.755742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55160)
0809 09:32:02.755874 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.755920 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260809 09:32:02.756124 15414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:37431: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
I20260809 09:32:02.759802 15283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } attempt: 1
I20260809 09:32:02.760448 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.760195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55166)
0809 09:32:02.760321 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.760378 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 09:32:02.761163 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.760955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55168)
0809 09:32:02.761063 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.761110 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 09:32:02.761307 15283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:37431: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
W20260809 09:32:02.803220 15422 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 09:32:02.803435 15422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:32:02.803475 15422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:32:02.803509 15422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:32:02.806988 15422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:32:02.807040 15422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:32:02.807058 15422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:32:02.807085 15422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:32:02.807133 15422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:32:02.811591 15422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:39877,127.9.102.61:45173,127.9.102.60:37431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.60.keytab
--principal=kudu/127.9.102.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.60:37431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.102.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:32:02.812481 15422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:32:02.813351 15422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:32:02.820976 15429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:32:02.820955 15433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:32:02.821058 15422 server_base.cc:1034] running on GCE node
W20260809 09:32:02.820971 15428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:32:02.821652 15422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:32:02.822567 15422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:32:02.823747 15422 hybrid_clock.cc:648] HybridClock initialized: now 1786267922823715 us; error 49 us; skew 500 ppm
I20260809 09:32:02.824306 15414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } attempt: 2
I20260809 09:32:02.824944 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.824697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55178)
0809 09:32:02.824819 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.824867 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260809 09:32:02.825584 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.825362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55194)
0809 09:32:02.825476 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.825519 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 09:32:02.825724 15414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:37431: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:32:02.826519 15422 init.cc:377] Logged in from keytab as kudu/127.9.102.60@KRBTEST.COM (short username kudu)
I20260809 09:32:02.827333 15422 webserver.cc:460] Webserver started at http://127.9.102.60:37411/ using document root <none> and password file <none>
I20260809 09:32:02.827821 15422 fs_manager.cc:359] Metadata directory not provided
I20260809 09:32:02.827904 15422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:32:02.828166 15422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:32:02.829972 15422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/data/instance:
uuid: "25f2226debcf4184b74e5e36eaa00715"
format_stamp: "Formatted at 2026-08-09 09:32:02 on dist-test-slave-7vjv"
server_key: "caf06634bdb9754ef97dd87e00af84d9"
server_key_iv: "f98f7ef0e1a5664e90ecfb533ad1c236"
server_key_version: "encryptionkey@0"
I20260809 09:32:02.830489 15422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/wal/instance:
uuid: "25f2226debcf4184b74e5e36eaa00715"
format_stamp: "Formatted at 2026-08-09 09:32:02 on dist-test-slave-7vjv"
server_key: "caf06634bdb9754ef97dd87e00af84d9"
server_key_iv: "f98f7ef0e1a5664e90ecfb533ad1c236"
server_key_version: "encryptionkey@0"
I20260809 09:32:02.834543 15422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260809 09:32:02.836448 15283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } attempt: 2
I20260809 09:32:02.837486 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.837141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55204)
0809 09:32:02.837337 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.837394 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260809 09:32:02.838337 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.838073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55212)
0809 09:32:02.838193 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.838263 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260809 09:32:02.838514 15283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:37431: Network error: Client connection negotiation failed: client connection to 127.9.102.60:37431: connect: Connection refused (error 111)
I20260809 09:32:02.844039 15449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:32:02.845466 15422 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.001s
I20260809 09:32:02.845583 15422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/wal
uuid: "25f2226debcf4184b74e5e36eaa00715"
format_stamp: "Formatted at 2026-08-09 09:32:02 on dist-test-slave-7vjv"
server_key: "caf06634bdb9754ef97dd87e00af84d9"
server_key_iv: "f98f7ef0e1a5664e90ecfb533ad1c236"
server_key_version: "encryptionkey@0"
I20260809 09:32:02.845692 15422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:32:02.880270 15422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:32:02.883409 15422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:32:02.883591 15422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:32:02.891495 15422 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.60:37431
I20260809 09:32:02.891772 15538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.60:37431 every 8 connection(s)
I20260809 09:32:02.892637 15422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
I20260809 09:32:02.895171 15539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:32:02.897990 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 15422
I20260809 09:32:02.898123 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786267646682796-9624-0/minicluster-data/master-2/wal/instance
I20260809 09:32:02.898377 9624 external_mini_cluster.cc:1428] Setting key e0da4c1e97935f64d357f2542a85aef3
I20260809 09:32:02.900904 15539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.60@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.60@KRBTEST.COM
I20260809 09:32:02.916374 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.903906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.9.102.62:39877)
0809 09:32:02.904057 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.904062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.904779 (+ 717us) server_negotiation.cc:407] Connection header received
0809 09:32:02.905905 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.905911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.905966 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.906055 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.907066 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.907565 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.908720 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.909028 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.909144 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.912161 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.912167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.912170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.912204 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.913865 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.914564 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.914568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.914570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.914654 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.915085 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.915091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.915095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.915366 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:02.915448 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:02.916007 (+ 559us) server_negotiation.cc:299] Negotiation successful
0809 09:32:02.916172 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 09:32:02.916915 15414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } attempt: 3
I20260809 09:32:02.917331 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.904789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.9.102.60:37431)
0809 09:32:02.905054 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.905062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.905091 (+ 29us) server_negotiation.cc:407] Connection header received
0809 09:32:02.905654 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.905672 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.905893 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.906110 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.907151 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.908731 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.909660 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.910124 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.910239 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.912510 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.912516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.912524 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.912547 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.914345 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.914830 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.914834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.914835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.914922 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.915191 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.915195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.915197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.915504 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:02.915726 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:02.916228 (+ 502us) server_negotiation.cc:299] Negotiation successful
0809 09:32:02.916441 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":119,"threads_started":1}
I20260809 09:32:02.917328 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.904025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:39877 (local address 127.0.0.1:59696)
0809 09:32:02.904587 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.904645 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:02.905650 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:02.906217 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:02.906222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:02.906668 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:02.906906 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.906918 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.907691 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.907694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.908592 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.908602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.909196 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.909201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.909432 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.910003 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:02.910025 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:02.911929 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:02.913980 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.913990 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.914002 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.914446 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.914757 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.914761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.914765 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.914955 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.915549 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:02.915557 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:02.915721 (+ 164us) client_negotiation.cc:769] Sending connection context
0809 09:32:02.916584 (+ 863us) client_negotiation.cc:240] Negotiation successful
0809 09:32:02.916816 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"spinlock_wait_cycles":1280,"thread_start_us":114,"threads_started":1}
I20260809 09:32:02.919073 15539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.61@KRBTEST.COM for kudu/127.9.102.60@KRBTEST.COM
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.60@KRBTEST.COM for kudu/127.9.102.61@KRBTEST.COM
I20260809 09:32:02.927266 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.917317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55238)
0809 09:32:02.917470 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.917489 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:02.917606 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:02.918267 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:02.918270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:02.918449 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:02.918557 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.918564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.920289 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.920295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.921148 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.921163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.921737 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.921740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.921813 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.922280 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:02.922296 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:02.923633 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:02.925612 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.925618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.925622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.925850 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.926186 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.926190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.926191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.926288 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.926737 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:02.926740 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:02.926834 (+ 94us) client_negotiation.cc:769] Sending connection context
0809 09:32:02.926923 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 09:32:02.927068 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":3}
I20260809 09:32:02.927893 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.917569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.9.102.60:37431)
0809 09:32:02.918060 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.918063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.918076 (+ 13us) server_negotiation.cc:407] Connection header received
0809 09:32:02.918108 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.918111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.918144 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.918241 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.918688 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.920140 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.921306 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.921623 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.921998 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.923806 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.923809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.923812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.923836 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.925483 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.925970 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.925973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.925974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.926060 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.926433 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.926436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.926438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.926631 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:02.927531 (+ 900us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:02.927614 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 09:32:02.927716 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":77,"threads_started":1}
I20260809 09:32:02.930402 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.920420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.9.102.62:39877)
0809 09:32:02.920564 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.920568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.920582 (+ 14us) server_negotiation.cc:407] Connection header received
0809 09:32:02.920626 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.920629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.920686 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.920769 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.921517 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.922161 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.923317 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.923660 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.923765 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.926713 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.926720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.926723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.926746 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.928434 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.929060 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.929064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.929065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.929148 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.929423 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.929426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.929427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.929617 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:02.929794 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:02.930166 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 09:32:02.930270 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 09:32:02.931100 15414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c: Bootstrap starting.
I20260809 09:32:02.932595 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.919721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.9.102.61:45173)
0809 09:32:02.919862 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.919866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.920024 (+ 158us) server_negotiation.cc:407] Connection header received
0809 09:32:02.920065 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.920071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.920152 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.920226 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.921057 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.921880 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.923540 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.923992 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.924868 (+ 876us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.927887 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.927893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.927899 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.927930 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.929750 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.930287 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.930292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.930297 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.930384 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.930806 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.930811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.930814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.931317 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:02.931661 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:02.932365 (+ 704us) server_negotiation.cc:299] Negotiation successful
0809 09:32:02.932469 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 09:32:02.932653 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.919483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:45173 (local address 127.0.0.1:50450)
0809 09:32:02.919835 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.919851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:02.919939 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:02.920542 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:02.920546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:02.920785 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:02.920913 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.920923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.922345 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.922351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.923372 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.923387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.924999 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.925004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.925129 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.925596 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:02.925614 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:02.927467 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:02.929879 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.929883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.929888 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.930169 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.930588 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.930594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.930597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.930688 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.931711 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:02.931715 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:02.932154 (+ 439us) client_negotiation.cc:769] Sending connection context
0809 09:32:02.932278 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 09:32:02.932417 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260809 09:32:02.933493 15539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:32:02.934175 15414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c: Neither blocks nor log segments found. Creating new log.
I20260809 09:32:02.935052 15414 log.cc:824] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c: Log is configured to *not* fsync() on all Append() calls
I20260809 09:32:02.937681 15283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } attempt: 3
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.61@KRBTEST.COM
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.60@KRBTEST.COM for kudu/127.9.102.60@KRBTEST.COM
I20260809 09:32:02.942984 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.931972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.9.102.61:45173)
0809 09:32:02.933088 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.933092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.933102 (+ 10us) server_negotiation.cc:407] Connection header received
0809 09:32:02.933131 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.933133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.933169 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.933993 (+ 824us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.934090 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.934871 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.935867 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.936059 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.936216 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.938655 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.938659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.938661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.938691 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.940741 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.941227 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.941234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.941240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.941322 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.941690 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.941694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.941697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.941919 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:02.942613 (+ 694us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:02.942691 (+ 78us) server_negotiation.cc:299] Negotiation successful
0809 09:32:02.942801 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1064,"thread_start_us":77,"threads_started":1}
Aug 09 09:32:02 dist-test-slave-7vjv krb5kdc[15125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267922, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.60@KRBTEST.COM
I20260809 09:32:02.944533 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.933954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55254)
0809 09:32:02.934161 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.934175 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:02.934268 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:02.934844 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:02.934850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:02.935042 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:02.935144 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.935152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.936314 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.936318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.937252 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.937264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.938349 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.938355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.938499 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.938967 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:02.938983 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:02.940653 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:02.942924 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.942930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.942931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.943224 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.943519 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.943522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.943523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.943608 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.944045 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:02.944048 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:02.944148 (+ 100us) client_negotiation.cc:769] Sending connection context
0809 09:32:02.944247 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 09:32:02.944379 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260809 09:32:02.944975 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.934416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.9.102.60:37431)
0809 09:32:02.934540 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.934550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.934564 (+ 14us) server_negotiation.cc:407] Connection header received
0809 09:32:02.934599 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.934604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.934650 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.934733 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.935272 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.936190 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.937387 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.937743 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.937845 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.940802 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.940807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.940813 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.940837 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.942584 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.943333 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.943336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.943338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.943423 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.943705 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.943708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.943709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.943891 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:02.944003 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:02.944709 (+ 706us) server_negotiation.cc:299] Negotiation successful
0809 09:32:02.944812 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 09:32:02.945976 15414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c: No bootstrap required, opened a new log
I20260809 09:32:02.947366 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:32:02.947984 15539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715: Bootstrap starting.
I20260809 09:32:02.949121 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.938258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.9.102.60:37431)
0809 09:32:02.938767 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.938772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.938788 (+ 16us) server_negotiation.cc:407] Connection header received
0809 09:32:02.938849 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.938853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.938902 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.939518 (+ 616us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.939654 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.940567 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.941853 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.942246 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.942884 (+ 638us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.944806 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.944809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.944811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.944829 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.947039 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.947586 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.947589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.947591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.947664 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.947987 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.947991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.947994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.948293 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:02.948829 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:02.948899 (+ 70us) server_negotiation.cc:299] Negotiation successful
0809 09:32:02.948995 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428}
I20260809 09:32:02.948758 15233 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "89541ebe94b64f38ab668de9d57c77c9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59712
I20260809 09:32:02.949975 15414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } }
I20260809 09:32:02.950176 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.938045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:37431 (local address 127.0.0.1:55270)
0809 09:32:02.938180 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:02.938193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:02.938261 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:02.939043 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:02.939048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:02.939269 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:02.939394 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.939403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.940710 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.940716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.941690 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:02.941703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.942389 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.942395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:02.942513 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.942949 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:02.942970 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:02.944559 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:02.947163 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.947166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.947167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.947419 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.947759 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:02.947762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:02.947764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:02.947873 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:02.948420 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:02.948425 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:02.948546 (+ 121us) client_negotiation.cc:769] Sending connection context
0809 09:32:02.948624 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 09:32:02.948752 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 09:32:02.950194 15414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215caf8187bc44adae3c7cbc33d8e27c, State: Initialized, Role: FOLLOWER
I20260809 09:32:02.950510 15414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } }
I20260809 09:32:02.950997 15548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } } }
I20260809 09:32:02.951112 15539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715: Neither blocks nor log segments found. Creating new log.
I20260809 09:32:02.951234 15548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:02.951861 15414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:32:02.952031 15539 log.cc:824] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715: Log is configured to *not* fsync() on all Append() calls
I20260809 09:32:02.952941 15283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9: Bootstrap starting.
I20260809 09:32:02.954051 15539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715: No bootstrap required, opened a new log
I20260809 09:32:02.954727 15283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9: Neither blocks nor log segments found. Creating new log.
I20260809 09:32:02.955421 15283 log.cc:824] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9: Log is configured to *not* fsync() on all Append() calls
I20260809 09:32:02.958797 15539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } }
I20260809 09:32:02.959048 15539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f2226debcf4184b74e5e36eaa00715, State: Initialized, Role: FOLLOWER
I20260809 09:32:02.959395 15539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } }
I20260809 09:32:02.960402 15575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } } }
I20260809 09:32:02.960752 15575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:02.960817 15539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:32:02.958710 15293 master_runner.cc:186] Error getting master registration for 127.9.102.62:39877: OK, instance_id { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" instance_seqno: 1786267922479479 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 09:32:02.962811 15293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 09:32:02.963037 15283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9: No bootstrap required, opened a new log
I20260809 09:32:02.966094 15283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } }
I20260809 09:32:02.966298 15283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89541ebe94b64f38ab668de9d57c77c9, State: Initialized, Role: FOLLOWER
W20260809 09:32:02.966491 15573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:32:02.966557 15573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:32:02.966598 15283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } }
W20260809 09:32:02.966964 15591 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:32:02.967082 15591 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:32:02.964768 15422 master_runner.cc:186] Error getting master registration for 127.9.102.62:39877: OK, instance_id { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" instance_seqno: 1786267922479479 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 09:32:02.967733 15422 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 09:32:02.967839 15283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:32:02.967975 15233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 09:32:02.968158 15233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:32:02.969278 15233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } }
I20260809 09:32:02.970687 15359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89541ebe94b64f38ab668de9d57c77c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "215caf8187bc44adae3c7cbc33d8e27c"
I20260809 09:32:02.970844 15499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "89541ebe94b64f38ab668de9d57c77c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "25f2226debcf4184b74e5e36eaa00715"
I20260809 09:32:02.970896 15359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:32:02.971045 15499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:32:02.972222 15499 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89541ebe94b64f38ab668de9d57c77c9 in term 1.
I20260809 09:32:02.972316 15359 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89541ebe94b64f38ab668de9d57c77c9 in term 1.
I20260809 09:32:02.972713 15194 leader_election.cc:302] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25f2226debcf4184b74e5e36eaa00715, 89541ebe94b64f38ab668de9d57c77c9; no voters:
I20260809 09:32:02.970144 15233 leader_election.cc:288] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [CANDIDATE]: Term 1 election: Requested vote from peers 215caf8187bc44adae3c7cbc33d8e27c (127.9.102.61:45173), 25f2226debcf4184b74e5e36eaa00715 (127.9.102.60:37431)
I20260809 09:32:02.973086 15596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } } }
I20260809 09:32:02.973323 15596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:02.973448 15596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:32:02.973588 15596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [term 1 LEADER]: Becoming Leader. State: Replica: 89541ebe94b64f38ab668de9d57c77c9, State: Running, Role: LEADER
I20260809 09:32:02.973922 15596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } }
I20260809 09:32:02.975778 15596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89541ebe94b64f38ab668de9d57c77c9. Latest consensus state: current_term: 1 leader_uuid: "89541ebe94b64f38ab668de9d57c77c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } } }
I20260809 09:32:02.975930 15596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [sys.catalog]: This master's current role is: LEADER
W20260809 09:32:02.976310 15200 tablet.cc:2369] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:32:02.977712 15150 master_runner.cc:263] No leader master found from master 89541ebe94b64f38ab668de9d57c77c9
I20260809 09:32:02.977780 15150 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 09:32:02.984583 15614 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:32:02.984670 15614 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:32:02.984771 15621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 09:32:02.985658 15621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:32:03.001940 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.981293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59718 (local address 127.9.102.62:39877)
0809 09:32:02.981415 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.981421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.983019 (+ 1598us) server_negotiation.cc:407] Connection header received
0809 09:32:02.985221 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.985227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.985284 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.985339 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.991072 (+ 5733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.991937 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.994972 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.995283 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.995370 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.997242 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.997245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.997248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.997272 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:02.999163 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:02.999676 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:02.999682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:02.999685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:02.999763 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:03.000836 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:03.000840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:03.000842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:03.001085 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:03.001210 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:03.001633 (+ 423us) server_negotiation.cc:299] Negotiation successful
0809 09:32:03.001754 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260809 09:32:03.004840 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.982002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.9.102.60:37431)
0809 09:32:02.982120 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.982126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.983248 (+ 1122us) server_negotiation.cc:407] Connection header received
0809 09:32:02.986517 (+ 3269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.986533 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.986590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.986647 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.990047 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.990919 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.995543 (+ 4624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.995850 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.995926 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:02.999626 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:02.999631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:02.999635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:02.999666 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:03.001438 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:03.001942 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:03.001947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:03.001952 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:03.002044 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:03.003798 (+ 1754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:03.003802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:03.003804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:03.004041 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:03.004191 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:03.004554 (+ 363us) server_negotiation.cc:299] Negotiation successful
0809 09:32:03.004677 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 09:32:03.006012 15499 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [term 1 FOLLOWER]: Refusing update from remote peer 89541ebe94b64f38ab668de9d57c77c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:32:03.006129 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:02.980509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.9.102.61:45173)
0809 09:32:02.980659 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:02.980665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:02.982773 (+ 2108us) server_negotiation.cc:407] Connection header received
0809 09:32:02.986787 (+ 4014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:02.986793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:02.986845 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:02.986898 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:02.988941 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.989895 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.996568 (+ 6673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:02.996961 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:02.997042 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:03.002423 (+ 5381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:03.002428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:03.002431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:03.002459 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:03.004280 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:03.004728 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:03.004731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:03.004733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:03.004809 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:03.005149 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:03.005156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:03.005160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:03.005388 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:03.005509 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:03.005849 (+ 340us) server_negotiation.cc:299] Negotiation successful
0809 09:32:03.005961 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 09:32:03.006723 15615 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:32:03.007254 15359 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [term 1 FOLLOWER]: Refusing update from remote peer 89541ebe94b64f38ab668de9d57c77c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:32:03.011065 15575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89541ebe94b64f38ab668de9d57c77c9. Latest consensus state: current_term: 1 leader_uuid: "89541ebe94b64f38ab668de9d57c77c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } } }
I20260809 09:32:03.011186 15575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:03.011304 15615 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:32:03.014899 15548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89541ebe94b64f38ab668de9d57c77c9. Latest consensus state: current_term: 1 leader_uuid: "89541ebe94b64f38ab668de9d57c77c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } } }
I20260809 09:32:03.015010 15548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:03.018908 15615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89541ebe94b64f38ab668de9d57c77c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } } }
I20260809 09:32:03.019021 15615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [sys.catalog]: This master's current role is: LEADER
I20260809 09:32:03.019134 15627 mvcc.cc:205] Tried to move back new op lower bound from 7316553412581982208 to 7316553412504903680. Current Snapshot: MvccSnapshot[applied={T|T < 7316553412581982208}]
I20260809 09:32:03.022014 15596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "89541ebe94b64f38ab668de9d57c77c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } } }
I20260809 09:32:03.022122 15596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9 [sys.catalog]: This master's current role is: LEADER
I20260809 09:32:03.023058 15548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89541ebe94b64f38ab668de9d57c77c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } } }
I20260809 09:32:03.023149 15548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:03.023331 15632 mvcc.cc:205] Tried to move back new op lower bound from 7316553412581982208 to 7316553412504903680. Current Snapshot: MvccSnapshot[applied={T|T < 7316553412581982208}]
I20260809 09:32:03.023794 15621 catalog_manager.cc:1347] Generated new cluster ID: 7d0beab2427e434cb890db058fbd37e3
I20260809 09:32:03.023849 15621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:32:03.028452 15628 mvcc.cc:205] Tried to move back new op lower bound from 7316553412581982208 to 7316553412504903680. Current Snapshot: MvccSnapshot[applied={T|T < 7316553412581982208}]
I20260809 09:32:03.029356 15575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "89541ebe94b64f38ab668de9d57c77c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89541ebe94b64f38ab668de9d57c77c9" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 39877 } } peers { permanent_uuid: "215caf8187bc44adae3c7cbc33d8e27c" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 45173 } } peers { permanent_uuid: "25f2226debcf4184b74e5e36eaa00715" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 37431 } } }
I20260809 09:32:03.029451 15575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:03.096467 15621 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:32:03.099162 15621 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:32:03.126837 15621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89541ebe94b64f38ab668de9d57c77c9: Generated new TSK 0
I20260809 09:32:03.127596 15621 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 09:32:03.184880 15543 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.182710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.9.102.60:37431)
0809 09:32:03.182822 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.182825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.183904 (+ 1079us) server_negotiation.cc:407] Connection header received
0809 09:32:03.183961 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.183965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.184035 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.184165 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.184737 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55298: BlockingRecv error: recv got EOF from 127.0.0.1:55298 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260809 09:32:03.184906 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.183109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50492 (local address 127.9.102.61:45173)
0809 09:32:03.183216 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.183219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.184026 (+ 807us) server_negotiation.cc:407] Connection header received
0809 09:32:03.184209 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.184212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.184256 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.184302 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.184790 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50492: BlockingRecv error: recv got EOF from 127.0.0.1:50492 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260809 09:32:03.185688 15290 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.182071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59730 (local address 127.9.102.62:39877)
0809 09:32:03.182222 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.182226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.183500 (+ 1274us) server_negotiation.cc:407] Connection header received
0809 09:32:03.183634 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.183637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.183685 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.183795 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.185587 (+ 1792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59730: BlockingRecv error: recv got EOF from 127.0.0.1:59730 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260809 09:32:03.188709 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.187535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.9.102.61:45173)
0809 09:32:03.187666 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.187669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.187981 (+ 312us) server_negotiation.cc:407] Connection header received
0809 09:32:03.188086 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.188118 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.188191 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.188275 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.188627 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50494: BlockingRecv error: recv got EOF from 127.0.0.1:50494 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260809 09:32:03.192619 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.191415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.9.102.61:45173)
0809 09:32:03.191493 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.191497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.191885 (+ 388us) server_negotiation.cc:407] Connection header received
0809 09:32:03.192038 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.192041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.192095 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.192168 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.192553 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50508: BlockingRecv error: recv got EOF from 127.0.0.1:50508 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260809 09:32:03.197553 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.196377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.9.102.61:45173)
0809 09:32:03.196455 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.196458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.196841 (+ 383us) server_negotiation.cc:407] Connection header received
0809 09:32:03.196988 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.196991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.197027 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.197068 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.197454 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50522: BlockingRecv error: recv got EOF from 127.0.0.1:50522 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260809 09:32:03.204818 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.203332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.9.102.61:45173)
0809 09:32:03.203459 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.203463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.203930 (+ 467us) server_negotiation.cc:407] Connection header received
0809 09:32:03.204095 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.204098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.204180 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.204229 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.204738 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50528: BlockingRecv error: recv got EOF from 127.0.0.1:50528 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 09:32:03.216017 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.214701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.9.102.61:45173)
0809 09:32:03.214794 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.214797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.215219 (+ 422us) server_negotiation.cc:407] Connection header received
0809 09:32:03.215366 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.215370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.215408 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.215447 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.215927 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50544: BlockingRecv error: recv got EOF from 127.0.0.1:50544 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260809 09:32:03.235038 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.233729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.9.102.61:45173)
0809 09:32:03.233827 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.233831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.234252 (+ 421us) server_negotiation.cc:407] Connection header received
0809 09:32:03.234412 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.234414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.234453 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.234492 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.234959 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50548: BlockingRecv error: recv got EOF from 127.0.0.1:50548 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260809 09:32:03.270551 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.269451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.9.102.61:45173)
0809 09:32:03.269597 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.269602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.269779 (+ 177us) server_negotiation.cc:407] Connection header received
0809 09:32:03.269881 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.269883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.269927 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.269998 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.270439 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50556: BlockingRecv error: recv got EOF from 127.0.0.1:50556 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260809 09:32:03.337520 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.336435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.9.102.61:45173)
0809 09:32:03.336584 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.336588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.336720 (+ 132us) server_negotiation.cc:407] Connection header received
0809 09:32:03.336873 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.336879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.336922 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.337001 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.337428 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50566: BlockingRecv error: recv got EOF from 127.0.0.1:50566 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260809 09:32:03.468644 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.467526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.9.102.61:45173)
0809 09:32:03.467649 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.467653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.467801 (+ 148us) server_negotiation.cc:407] Connection header received
0809 09:32:03.467906 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.467909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.467953 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.468024 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.468487 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50572: BlockingRecv error: recv got EOF from 127.0.0.1:50572 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 09:32:03.728132 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:03.726895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.9.102.61:45173)
0809 09:32:03.727026 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:03.727031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:03.727144 (+ 113us) server_negotiation.cc:407] Connection header received
0809 09:32:03.727350 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:03.727355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:03.727409 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:03.727499 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:03.727977 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50588: BlockingRecv error: recv got EOF from 127.0.0.1:50588 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 09:32:03.967795 15573 catalog_manager.cc:1259] Loaded cluster ID: 7d0beab2427e434cb890db058fbd37e3
I20260809 09:32:03.967866 15573 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c: loading cluster ID for follower catalog manager: success
I20260809 09:32:03.968286 15591 catalog_manager.cc:1259] Loaded cluster ID: 7d0beab2427e434cb890db058fbd37e3
I20260809 09:32:03.968354 15591 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715: loading cluster ID for follower catalog manager: success
I20260809 09:32:03.969762 15573 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c: acquiring CA information for follower catalog manager: success
I20260809 09:32:03.969995 15591 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715: acquiring CA information for follower catalog manager: success
I20260809 09:32:03.970909 15573 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 215caf8187bc44adae3c7cbc33d8e27c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 09:32:03.970949 15591 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25f2226debcf4184b74e5e36eaa00715: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 09:32:04.245884 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:04.242092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.9.102.61:45173)
0809 09:32:04.242318 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:04.242323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:04.242398 (+ 75us) server_negotiation.cc:407] Connection header received
0809 09:32:04.242571 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:04.242574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:04.242705 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:04.242825 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:32:04.243083 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:04.243563 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:04.244415 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:04.244629 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:04.244734 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:04.245038 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:32:04.245478 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:32:04.245618 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:04.245677 (+ 59us) server_negotiation.cc:299] Negotiation successful
0809 09:32:04.245730 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":72,"threads_started":1}
I20260809 09:32:04.252285 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:04.249001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.9.102.60:37431)
0809 09:32:04.249226 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:04.249229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:04.249242 (+ 13us) server_negotiation.cc:407] Connection header received
0809 09:32:04.249345 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:04.249347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:04.249447 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:04.249538 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:32:04.249708 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:04.250178 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:04.250978 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:04.251176 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:04.251244 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:04.251477 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:32:04.251888 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:32:04.251962 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:04.252056 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 09:32:04.252142 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260809 09:32:04.258325 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:04.254832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.9.102.61:45173)
0809 09:32:04.254912 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:04.254916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:04.255060 (+ 144us) server_negotiation.cc:407] Connection header received
0809 09:32:04.255173 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:04.255176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:04.255215 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:04.255290 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:32:04.255576 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:04.256216 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:04.257313 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:04.257553 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:04.257666 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:04.257914 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:32:04.257983 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:32:04.258075 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:04.258155 (+ 80us) server_negotiation.cc:299] Negotiation successful
0809 09:32:04.258197 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260809 09:32:04.258966 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:04.254823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.9.102.60:37431)
0809 09:32:04.254923 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:04.254927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:04.255279 (+ 352us) server_negotiation.cc:407] Connection header received
0809 09:32:04.255421 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:04.255426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:04.255479 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:04.255537 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:32:04.255855 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:04.256678 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:04.257724 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:04.258056 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:04.258140 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:04.258422 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:32:04.258480 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:32:04.258553 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:04.258817 (+ 264us) server_negotiation.cc:299] Negotiation successful
0809 09:32:04.258868 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260809 09:32:04.259624 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:04.254656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.9.102.62:39877)
0809 09:32:04.255263 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:04.255268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:04.255492 (+ 224us) server_negotiation.cc:407] Connection header received
0809 09:32:04.255625 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:04.255628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:04.255759 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:04.255858 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:32:04.256224 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:04.256984 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:04.258161 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:04.258482 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:04.258574 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:04.258939 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:32:04.259215 (+ 276us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:32:04.259316 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:04.259413 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 09:32:04.259457 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":54,"threads_started":1}
I20260809 09:32:04.262991 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 15150
I20260809 09:32:04.281417 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 15293
I20260809 09:32:04.293649 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 15422
2026-08-09T09:32:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3264 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 09:32:04 dist-test-slave-7vjv krb5kdc[15778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 09:32:04 dist-test-slave-7vjv krb5kdc[15778](info): set up 2 sockets
Aug 09 09:32:04 dist-test-slave-7vjv krb5kdc[15778](info): commencing operation
krb5kdc: starting...
W20260809 09:32:05.446215 9624 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/test-user.keytab.
Aug 09 09:32:05 dist-test-slave-7vjv krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T09:32:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T09:32:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
Entry for principal kudu/127.9.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab.
I20260809 09:32:05.556272 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:43437
--webserver_interface=127.9.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:43437,127.9.102.61:36479,127.9.102.60:44991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 09:32:05.655081 15794 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 09:32:05.655284 15794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:32:05.655303 15794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:32:05.655332 15794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:32:05.658900 15794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:32:05.658946 15794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:32:05.658963 15794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:32:05.658977 15794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:32:05.658991 15794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:32:05.663371 15794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:43437,127.9.102.61:36479,127.9.102.60:44991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.62.keytab
--principal=kudu/127.9.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.62:43437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:32:05.664215 15794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:32:05.665028 15794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:32:05.672302 15802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:32:05.672267 15804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:32:05.672431 15794 server_base.cc:1034] running on GCE node
W20260809 09:32:05.672283 15806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:32:05.672940 15794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:32:05.673751 15794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:32:05.674899 15794 hybrid_clock.cc:648] HybridClock initialized: now 1786267925674883 us; error 27 us; skew 500 ppm
Aug 09 09:32:05 dist-test-slave-7vjv krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267925, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:32:05.677506 15794 init.cc:377] Logged in from keytab as kudu/127.9.102.62@KRBTEST.COM (short username kudu)
I20260809 09:32:05.678267 15794 webserver.cc:460] Webserver started at http://127.9.102.62:39593/ using document root <none> and password file <none>
I20260809 09:32:05.678795 15794 fs_manager.cc:359] Metadata directory not provided
I20260809 09:32:05.678880 15794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:32:05.679086 15794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:32:05.680859 15794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/data/instance:
uuid: "825c670335614840b8daef4763015bf2"
format_stamp: "Formatted at 2026-08-09 09:32:05 on dist-test-slave-7vjv"
server_key: "526ed821dac7dc83acd0c6042d3f5a87"
server_key_iv: "5deaa8ef3b387277d9fb3db0bb6c1151"
server_key_version: "encryptionkey@0"
I20260809 09:32:05.681391 15794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/wal/instance:
uuid: "825c670335614840b8daef4763015bf2"
format_stamp: "Formatted at 2026-08-09 09:32:05 on dist-test-slave-7vjv"
server_key: "526ed821dac7dc83acd0c6042d3f5a87"
server_key_iv: "5deaa8ef3b387277d9fb3db0bb6c1151"
server_key_version: "encryptionkey@0"
I20260809 09:32:05.684557 15794 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:32:05.686674 15817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:32:05.687711 15794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:32:05.687842 15794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/wal
uuid: "825c670335614840b8daef4763015bf2"
format_stamp: "Formatted at 2026-08-09 09:32:05 on dist-test-slave-7vjv"
server_key: "526ed821dac7dc83acd0c6042d3f5a87"
server_key_iv: "5deaa8ef3b387277d9fb3db0bb6c1151"
server_key_version: "encryptionkey@0"
I20260809 09:32:05.687940 15794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:32:05.701663 15794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:32:05.704213 15794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:32:05.704406 15794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:32:05.710564 15794 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.62:43437
I20260809 09:32:05.710841 15904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.62:43437 every 8 connection(s)
I20260809 09:32:05.711428 15794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/data/info.pb
I20260809 09:32:05.713798 15906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:32:05.717125 15906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:32:05.721339 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 15794
I20260809 09:32:05.721472 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-0/wal/instance
I20260809 09:32:05.721773 9624 external_mini_cluster.cc:1428] Setting key 7844f20bf0edf6a986faec2e071570ad
Aug 09 09:32:05 dist-test-slave-7vjv krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267925, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.102.61@KRBTEST.COM; defaulting to no policy
I20260809 09:32:05.731142 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.719069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:43437 (local address 127.0.0.1:39626)
0809 09:32:05.719663 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.719698 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:05.720514 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:05.721166 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:05.721170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:05.721594 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:05.721832 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:05.721863 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.722928 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.722931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:05.723779 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:05.723793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.724453 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.724459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:05.724599 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:05.725110 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:05.725133 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:05.726665 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:05.728496 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:05.728505 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:05.728514 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:05.728730 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:05.729038 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:05.729041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:05.729043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:05.729149 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:05.729657 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:05.729662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:05.729785 (+ 123us) client_negotiation.cc:769] Sending connection context
0809 09:32:05.729970 (+ 185us) client_negotiation.cc:240] Negotiation successful
0809 09:32:05.730331 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":110,"spinlock_wait_cycles":48384,"thread_start_us":169,"threads_started":1}
I20260809 09:32:05.731142 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.719069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39626 (local address 127.9.102.62:43437)
0809 09:32:05.719550 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:05.719582 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:05.719791 (+ 209us) server_negotiation.cc:407] Connection header received
0809 09:32:05.720800 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:05.720805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:05.720974 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:05.721169 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:05.721988 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.722806 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.723909 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.724309 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.724422 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:05.726802 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:05.726807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:05.726815 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:05.726831 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:05.728377 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:05.728833 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:05.728838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:05.728840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:05.728939 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:05.729241 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:05.729244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:05.729246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:05.729553 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:05.729678 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:05.730029 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 09:32:05.730270 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":326,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.61@KRBTEST.COM" created.
I20260809 09:32:05.733742 15906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:32:05.734494 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.734194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:36479 (local address 127.0.0.1:41254)
0809 09:32:05.734330 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.734416 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 09:32:05.734719 15821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.102.61:36479: connect: Connection refused (error 111)
I20260809 09:32:05.735668 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.735425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:36479 (local address 127.0.0.1:41264)
0809 09:32:05.735577 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.735619 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260809 09:32:05.735901 15906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.61:36479: Network error: Client connection negotiation failed: client connection to 127.9.102.61:36479: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.61.keytab.
Entry for principal kudu/127.9.102.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.61.keytab.
I20260809 09:32:05.746842 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.61:36479
--webserver_interface=127.9.102.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.61.keytab
--principal=kudu/127.9.102.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:43437,127.9.102.61:36479,127.9.102.60:44991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:32:05.785135 15906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } attempt: 1
I20260809 09:32:05.786059 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.785776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:36479 (local address 127.0.0.1:41278)
0809 09:32:05.785927 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.785987 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 09:32:05.786751 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.786543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:36479 (local address 127.0.0.1:41284)
0809 09:32:05.786648 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.786697 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 09:32:05.786871 15906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.61:36479: Network error: Client connection negotiation failed: client connection to 127.9.102.61:36479: connect: Connection refused (error 111)
W20260809 09:32:05.847419 15913 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 09:32:05.847681 15913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:32:05.847743 15913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:32:05.847807 15913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:32:05.851549 15913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:32:05.851624 15913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:32:05.851663 15913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:32:05.851696 15913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:32:05.851723 15913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:32:05.855010 15906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } attempt: 2
I20260809 09:32:05.855664 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.855403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:36479 (local address 127.0.0.1:41300)
0809 09:32:05.855541 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.855603 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 09:32:05.856248 15913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:43437,127.9.102.61:36479,127.9.102.60:44991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.61.keytab
--principal=kudu/127.9.102.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.61:36479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.102.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:32:05.856319 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.856148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:36479 (local address 127.0.0.1:41310)
0809 09:32:05.856226 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.856266 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.61:36479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260809 09:32:05.856519 15906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.61:36479: Network error: Client connection negotiation failed: client connection to 127.9.102.61:36479: connect: Connection refused (error 111)
I20260809 09:32:05.857117 15913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:32:05.857944 15913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:32:05.864992 15923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:32:05.864972 15922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:32:05.864976 15925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:32:05.865358 15913 server_base.cc:1034] running on GCE node
I20260809 09:32:05.865844 15913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:32:05.866675 15913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:32:05.867830 15913 hybrid_clock.cc:648] HybridClock initialized: now 1786267925867814 us; error 25 us; skew 500 ppm
Aug 09 09:32:05 dist-test-slave-7vjv krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267925, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:32:05.870340 15913 init.cc:377] Logged in from keytab as kudu/127.9.102.61@KRBTEST.COM (short username kudu)
I20260809 09:32:05.871052 15913 webserver.cc:460] Webserver started at http://127.9.102.61:44983/ using document root <none> and password file <none>
I20260809 09:32:05.871510 15913 fs_manager.cc:359] Metadata directory not provided
I20260809 09:32:05.871608 15913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:32:05.871819 15913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:32:05.873543 15913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/data/instance:
uuid: "07f8e1f8ed494e02a022ac5b9651d34d"
format_stamp: "Formatted at 2026-08-09 09:32:05 on dist-test-slave-7vjv"
server_key: "be4ad727d559b0f1083021b6c58f76bb"
server_key_iv: "bc0b2dd2696e6f2ff2d3ef821fb3215c"
server_key_version: "encryptionkey@0"
I20260809 09:32:05.874049 15913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/wal/instance:
uuid: "07f8e1f8ed494e02a022ac5b9651d34d"
format_stamp: "Formatted at 2026-08-09 09:32:05 on dist-test-slave-7vjv"
server_key: "be4ad727d559b0f1083021b6c58f76bb"
server_key_iv: "bc0b2dd2696e6f2ff2d3ef821fb3215c"
server_key_version: "encryptionkey@0"
I20260809 09:32:05.877182 15913 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 09:32:05.879215 15940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:32:05.880321 15913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:32:05.880443 15913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/wal
uuid: "07f8e1f8ed494e02a022ac5b9651d34d"
format_stamp: "Formatted at 2026-08-09 09:32:05 on dist-test-slave-7vjv"
server_key: "be4ad727d559b0f1083021b6c58f76bb"
server_key_iv: "bc0b2dd2696e6f2ff2d3ef821fb3215c"
server_key_version: "encryptionkey@0"
I20260809 09:32:05.880537 15913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:32:05.892318 15913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:32:05.894757 15913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:32:05.894929 15913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:32:05.900996 15913 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.61:36479
I20260809 09:32:05.901288 16036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.61:36479 every 8 connection(s)
I20260809 09:32:05.901834 15913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/data/info.pb
I20260809 09:32:05.902262 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 15913
I20260809 09:32:05.902393 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-1/wal/instance
I20260809 09:32:05.902709 9624 external_mini_cluster.cc:1428] Setting key 9460fd0dff739adb221a0b9cefa55c91
I20260809 09:32:05.904883 16040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:32:05.908088 16040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.102.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.102.60@KRBTEST.COM" created.
Aug 09 09:32:05 dist-test-slave-7vjv krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267925, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.61@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:32:05.920972 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.909712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.9.102.62:43437)
0809 09:32:05.909818 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:05.909821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:05.910694 (+ 873us) server_negotiation.cc:407] Connection header received
0809 09:32:05.911599 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:05.911603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:05.911665 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:05.911766 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:05.912876 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.913365 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.914460 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.914858 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.914950 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:05.917451 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:05.917454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:05.917456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:05.917475 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:05.918917 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:05.919440 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:05.919444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:05.919445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:05.919523 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:05.919860 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:05.919864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:05.919866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:05.920020 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:05.920197 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:05.920687 (+ 490us) server_negotiation.cc:299] Negotiation successful
0809 09:32:05.920820 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 09:32:05.921655 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.910115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:43437 (local address 127.0.0.1:39642)
0809 09:32:05.910565 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.910602 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:05.911395 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:05.911924 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:05.911930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:05.912304 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:05.912736 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:05.912749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.913512 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.913518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:05.914333 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:05.914343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.915030 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.915037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:05.915241 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:05.915746 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:05.915770 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:05.917296 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:05.919037 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:05.919062 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:05.919074 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:05.919339 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:05.919626 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:05.919629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:05.919631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:05.919760 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:05.920195 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:05.920201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:05.920417 (+ 216us) client_negotiation.cc:769] Sending connection context
0809 09:32:05.920612 (+ 195us) client_negotiation.cc:240] Negotiation successful
0809 09:32:05.920922 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":139,"threads_started":1}
I20260809 09:32:05.923005 16040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 09:32:05 dist-test-slave-7vjv krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267925, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.61@KRBTEST.COM for kudu/127.9.102.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.102.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.60.keytab.
Entry for principal kudu/127.9.102.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.60.keytab.
I20260809 09:32:05.928591 9624 external_mini_cluster.cc:1326] Running /tmp/dist-test-taska20N0I/build/debug/bin/kudu
/tmp/dist-test-taska20N0I/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.60:44991
--webserver_interface=127.9.102.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.60.keytab
--principal=kudu/127.9.102.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.102.20:34303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.102.62:43437,127.9.102.61:36479,127.9.102.60:44991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 09:32:05.931226 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.923447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:36479 (local address 127.0.0.1:41316)
0809 09:32:05.923586 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.923597 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:05.923656 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:05.924304 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:05.924307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:05.924458 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:05.924540 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:05.924545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.925528 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.925531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:05.926015 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:05.926024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.926570 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.926572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:05.926634 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:05.926986 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:05.926995 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:05.928037 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:05.929916 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:05.929920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:05.929922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:05.930091 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:05.930348 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:05.930350 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:05.930352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:05.930421 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:05.930799 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:05.930801 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:05.930870 (+ 69us) client_negotiation.cc:769] Sending connection context
0809 09:32:05.930969 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 09:32:05.931092 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 09:32:05.931358 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.923564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.9.102.61:36479)
0809 09:32:05.923867 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:05.923876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:05.923919 (+ 43us) server_negotiation.cc:407] Connection header received
0809 09:32:05.923963 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:05.923967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:05.924173 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:05.924292 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:05.924640 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.925403 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.926117 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.926473 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.926548 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:05.928217 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:05.928225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:05.928234 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:05.928253 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:05.929801 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:05.930185 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:05.930190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:05.930191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:05.930259 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:05.930495 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:05.930499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:05.930500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:05.930696 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:05.930817 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:05.931078 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 09:32:05.931184 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":139,"threads_started":1}
I20260809 09:32:05.932976 16040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:32:05.933561 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.933351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59840)
0809 09:32:05.933443 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.933498 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 09:32:05.933730 15946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
I20260809 09:32:05.934496 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.934312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59842)
0809 09:32:05.934405 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.934447 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260809 09:32:05.934643 16040 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:44991: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
I20260809 09:32:05.947765 15906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } attempt: 3
Aug 09 09:32:05 dist-test-slave-7vjv krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267925, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.61@KRBTEST.COM
I20260809 09:32:05.955215 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.948209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:36479 (local address 127.0.0.1:41322)
0809 09:32:05.948330 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.948343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:05.948421 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:05.948685 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:05.948688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:05.948847 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:05.948921 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:05.948926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.949630 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.949637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:05.950202 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:05.950211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.950656 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.950659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:05.950770 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:05.951171 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:05.951182 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:05.952375 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:05.953836 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:05.953841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:05.953842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:05.954018 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:05.954273 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:05.954275 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:05.954277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:05.954355 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:05.954723 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:05.954726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:05.954825 (+ 99us) client_negotiation.cc:769] Sending connection context
0809 09:32:05.954933 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 09:32:05.955061 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 09:32:05.955227 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.948267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.9.102.61:36479)
0809 09:32:05.948371 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:05.948377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:05.948390 (+ 13us) server_negotiation.cc:407] Connection header received
0809 09:32:05.948519 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:05.948521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:05.948562 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:05.948668 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:05.949029 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.949488 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.950318 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:05.950547 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:05.950654 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:05.952493 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:05.952496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:05.952498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:05.952523 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:05.953725 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:05.954117 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:05.954120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:05.954122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:05.954190 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:05.954442 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:05.954445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:05.954446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:05.954608 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:05.954730 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:05.954996 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 09:32:05.955108 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 09:32:05.956619 15906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:32:05.957202 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.957016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59856)
0809 09:32:05.957118 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.957158 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 09:32:05.957803 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.957604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59866)
0809 09:32:05.957718 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.957759 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260809 09:32:05.957929 15906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:44991: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
I20260809 09:32:05.983855 16040 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } attempt: 1
I20260809 09:32:05.984575 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.984318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59876)
0809 09:32:05.984440 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.984500 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 09:32:05.985253 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.985051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59892)
0809 09:32:05.985153 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.985203 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 09:32:05.985410 16040 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:44991: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
I20260809 09:32:05.989086 15906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } attempt: 1
I20260809 09:32:05.989634 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.989439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59894)
0809 09:32:05.989538 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.989585 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 09:32:05.990180 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:05.989995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59910)
0809 09:32:05.990085 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:05.990128 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260809 09:32:05.990289 15906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:44991: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
W20260809 09:32:06.032380 16047 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 09:32:06.032645 16047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 09:32:06.032709 16047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 09:32:06.032778 16047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 09:32:06.036559 16047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 09:32:06.036634 16047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 09:32:06.036670 16047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 09:32:06.036702 16047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 09:32:06.036731 16047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 09:32:06.041383 16047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.102.20:34303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.102.62:43437,127.9.102.61:36479,127.9.102.60:44991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/krb5kdc/kudu_127.9.102.60.keytab
--principal=kudu/127.9.102.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.102.60:44991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.102.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 09 Aug 2026 08:38:55 UTC on ccycloud.quasar-njjleu.root.comops.site
build id 4575935
I20260809 09:32:06.042351 16047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 09:32:06.043208 16047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 09:32:06.050662 16062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:32:06.050704 16060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 09:32:06.050693 16065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 09:32:06.050879 16047 server_base.cc:1034] running on GCE node
I20260809 09:32:06.051373 16047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 09:32:06.052356 16047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 09:32:06.053541 16047 hybrid_clock.cc:648] HybridClock initialized: now 1786267926053512 us; error 30 us; skew 500 ppm
I20260809 09:32:06.053581 16040 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } attempt: 2
I20260809 09:32:06.054353 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.054024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59922)
0809 09:32:06.054199 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:06.054261 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 09:32:06.055112 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.054868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59938)
0809 09:32:06.054968 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:06.055025 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 09:32:06.055241 16040 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:44991: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Aug 09 09:32:06 dist-test-slave-7vjv krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267926, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 09:32:06.056667 16047 init.cc:377] Logged in from keytab as kudu/127.9.102.60@KRBTEST.COM (short username kudu)
I20260809 09:32:06.057417 16047 webserver.cc:460] Webserver started at http://127.9.102.60:41283/ using document root <none> and password file <none>
I20260809 09:32:06.057930 16047 fs_manager.cc:359] Metadata directory not provided
I20260809 09:32:06.058048 16047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 09:32:06.058260 16047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 09:32:06.060011 16047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/data/instance:
uuid: "e9f0aa934cd449e4940f962aa8e1b8f3"
format_stamp: "Formatted at 2026-08-09 09:32:06 on dist-test-slave-7vjv"
server_key: "77d243a5b345b7e0417543a84b669440"
server_key_iv: "26a40d58e07bf640418af34cab21c9f1"
server_key_version: "encryptionkey@0"
I20260809 09:32:06.060591 16047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/wal/instance:
uuid: "e9f0aa934cd449e4940f962aa8e1b8f3"
format_stamp: "Formatted at 2026-08-09 09:32:06 on dist-test-slave-7vjv"
server_key: "77d243a5b345b7e0417543a84b669440"
server_key_iv: "26a40d58e07bf640418af34cab21c9f1"
server_key_version: "encryptionkey@0"
I20260809 09:32:06.063736 16047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 09:32:06.065435 15906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } attempt: 2
I20260809 09:32:06.066051 16077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 09:32:06.066082 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.065846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59950)
0809 09:32:06.065971 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:06.066016 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 09:32:06.066808 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.066600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59956)
0809 09:32:06.066700 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:06.066747 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 09:32:06.066932 15906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.102.60:44991: Network error: Client connection negotiation failed: client connection to 127.9.102.60:44991: connect: Connection refused (error 111)
I20260809 09:32:06.067273 16047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260809 09:32:06.067366 16047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/data,/tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/wal
uuid: "e9f0aa934cd449e4940f962aa8e1b8f3"
format_stamp: "Formatted at 2026-08-09 09:32:06 on dist-test-slave-7vjv"
server_key: "77d243a5b345b7e0417543a84b669440"
server_key_iv: "26a40d58e07bf640418af34cab21c9f1"
server_key_version: "encryptionkey@0"
I20260809 09:32:06.067435 16047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 09:32:06.083565 16047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 09:32:06.086117 16047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 09:32:06.086258 16047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 09:32:06.092815 16047 rpc_server.cc:307] RPC server started. Bound to: 127.9.102.60:44991
I20260809 09:32:06.093079 16167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.102.60:44991 every 8 connection(s)
I20260809 09:32:06.093724 16047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/data/info.pb
I20260809 09:32:06.096199 16175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 09:32:06.099318 16175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:32:06.103801 9624 external_mini_cluster.cc:1388] Started /tmp/dist-test-taska20N0I/build/debug/bin/kudu as pid 16047
I20260809 09:32:06.103912 9624 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taska20N0I/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786267646682796-9624-0/minicluster-data/master-2/wal/instance
I20260809 09:32:06.104300 9624 external_mini_cluster.cc:1428] Setting key 5df8698f996f9dca6b5f6982614cbe6a
Aug 09 09:32:06 dist-test-slave-7vjv krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267926, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.60@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
Aug 09 09:32:06 dist-test-slave-7vjv krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.60@KRBTEST.COM
I20260809 09:32:06.111629 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.101153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.9.102.62:43437)
0809 09:32:06.101289 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.101293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.101717 (+ 424us) server_negotiation.cc:407] Connection header received
0809 09:32:06.102447 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.102450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.102491 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.102566 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.103540 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.104081 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.105416 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.105721 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.105819 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.107908 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:06.107911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:06.107913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:06.107949 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:06.109425 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.109916 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.109919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.109921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.110020 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.110381 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.110387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.110390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.110617 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:06.110695 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:06.111272 (+ 577us) server_negotiation.cc:299] Negotiation successful
0809 09:32:06.111429 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 09:32:06.112485 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.101127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.62:43437 (local address 127.0.0.1:39648)
0809 09:32:06.101588 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:06.101623 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:06.102273 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:06.102702 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:06.102708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:06.103041 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:06.103424 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:06.103435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.104225 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.104229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:06.105330 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:06.105340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.105862 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.105868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:06.106061 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.106583 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:06.106597 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:06.107766 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:06.109551 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:06.109559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:06.109581 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:06.109816 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:06.110149 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:06.110152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:06.110154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:06.110282 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:06.110717 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:06.110723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:06.111000 (+ 277us) client_negotiation.cc:769] Sending connection context
0809 09:32:06.111213 (+ 213us) client_negotiation.cc:240] Negotiation successful
0809 09:32:06.111506 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":119,"threads_started":1}
I20260809 09:32:06.113827 16175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:32:06.114594 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.104947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.9.102.60:44991)
0809 09:32:06.105291 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.105301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.105324 (+ 23us) server_negotiation.cc:407] Connection header received
0809 09:32:06.105367 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.105372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.105524 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.105659 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.106456 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.107078 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.107900 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.108264 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.108368 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.110601 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:06.110607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:06.110615 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:06.110633 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:06.112392 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.113283 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.113290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.113292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.113381 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.113642 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.113645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.113647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.113883 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:06.114076 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:06.114295 (+ 219us) server_negotiation.cc:299] Negotiation successful
0809 09:32:06.114415 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":145,"threads_started":1}
Aug 09 09:32:06 dist-test-slave-7vjv krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267926, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.60@KRBTEST.COM for kudu/127.9.102.61@KRBTEST.COM
Aug 09 09:32:06 dist-test-slave-7vjv krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.62@KRBTEST.COM
I20260809 09:32:06.121834 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.114244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.61:36479 (local address 127.0.0.1:41338)
0809 09:32:06.114380 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:06.114392 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:06.114457 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:06.114728 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:06.114732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:06.114888 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:06.115008 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:06.115013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.115797 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.115800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:06.116497 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:06.116507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.116980 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.116986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:06.117096 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.117433 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:06.117442 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:06.118596 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:06.120383 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:06.120386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:06.120387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:06.120614 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:06.120882 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:06.120885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:06.120887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:06.120964 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:06.121363 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:06.121365 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:06.121467 (+ 102us) client_negotiation.cc:769] Sending connection context
0809 09:32:06.121573 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 09:32:06.121688 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260809 09:32:06.121922 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.114331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.9.102.61:36479)
0809 09:32:06.114442 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.114445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.114458 (+ 13us) server_negotiation.cc:407] Connection header received
0809 09:32:06.114547 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.114550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.114604 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.114694 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.115145 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.115655 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.116601 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.116856 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.116939 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.118732 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:06.118735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:06.118737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:06.118761 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:06.120286 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.120716 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.120719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.120721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.120800 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.121082 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.121085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.121087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.121265 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:06.121358 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:06.121651 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 09:32:06.121761 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 09:32:06.122617 16175 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } has no permanent_uuid. Determining permanent_uuid...
I20260809 09:32:06.124382 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.116354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.9.102.62:43437)
0809 09:32:06.116440 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.116444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.116463 (+ 19us) server_negotiation.cc:407] Connection header received
0809 09:32:06.116586 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.116589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.116622 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.116688 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.117439 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.117956 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.118724 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.118925 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.119016 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.121063 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:06.121066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:06.121068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:06.121088 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:06.122677 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.123185 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.123190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.123191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.123283 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.123562 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.123565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.123566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.123743 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:06.123982 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:06.124084 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 09:32:06.124248 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Aug 09 09:32:06 dist-test-slave-7vjv krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267926, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.60@KRBTEST.COM for kudu/127.9.102.60@KRBTEST.COM
Aug 09 09:32:06 dist-test-slave-7vjv krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.102.61@KRBTEST.COM
I20260809 09:32:06.130648 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.122981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59972)
0809 09:32:06.123097 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:06.123109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:06.123189 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:06.123390 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:06.123392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:06.123537 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:06.123617 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:06.123623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.124636 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.124639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:06.125261 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:06.125272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.125750 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.125753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:06.125821 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.126279 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:06.126294 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:06.127347 (+ 1053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:06.129239 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:06.129242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:06.129243 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:06.129420 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:06.129740 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:06.129745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:06.129750 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:06.129854 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:06.130197 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:06.130200 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:06.130278 (+ 78us) client_negotiation.cc:769] Sending connection context
0809 09:32:06.130375 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 09:32:06.130491 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 09:32:06.130654 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.123062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.9.102.60:44991)
0809 09:32:06.123178 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.123181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.123191 (+ 10us) server_negotiation.cc:407] Connection header received
0809 09:32:06.123263 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.123265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.123301 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.123369 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.123720 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.124518 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.125407 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.125643 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.125740 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.127469 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:06.127472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:06.127475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:06.127501 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:06.129116 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.129522 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.129525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.129527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.129619 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.129957 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.129958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.129959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.130099 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:06.130205 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:06.130417 (+ 212us) server_negotiation.cc:299] Negotiation successful
0809 09:32:06.130542 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":44}
I20260809 09:32:06.133412 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.125379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.9.102.61:36479)
0809 09:32:06.125489 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.125492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.125504 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:32:06.125625 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.125628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.125664 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.125745 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.126304 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.126801 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.127577 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.127843 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.127951 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.130506 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:06.130509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:06.130511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:06.130533 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:06.131961 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.132379 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.132384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.132386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.132477 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.132740 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.132742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.132744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.132926 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:06.133125 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:06.133193 (+ 68us) server_negotiation.cc:299] Negotiation successful
0809 09:32:06.133297 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 09:32:06.134145 16175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3: Bootstrap starting.
I20260809 09:32:06.136288 9624 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 09:32:06.136512 16175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3: Neither blocks nor log segments found. Creating new log.
I20260809 09:32:06.137239 16175 log.cc:824] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3: Log is configured to *not* fsync() on all Append() calls
I20260809 09:32:06.137311 15861 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "825c670335614840b8daef4763015bf2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39650
I20260809 09:32:06.139104 16175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3: No bootstrap required, opened a new log
I20260809 09:32:06.142553 16175 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } }
I20260809 09:32:06.142766 16175 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f0aa934cd449e4940f962aa8e1b8f3, State: Initialized, Role: FOLLOWER
I20260809 09:32:06.143062 16175 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } }
I20260809 09:32:06.143445 16184 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } } }
I20260809 09:32:06.143573 16184 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:06.144227 16175 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:32:06.146384 16040 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } attempt: 3
W20260809 09:32:06.148905 16207 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:32:06.149041 16207 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 09 09:32:06 dist-test-slave-7vjv krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267925, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.61@KRBTEST.COM for kudu/127.9.102.60@KRBTEST.COM
I20260809 09:32:06.147907 16047 master_runner.cc:186] Error getting master registration for 127.9.102.62:43437: OK, instance_id { permanent_uuid: "825c670335614840b8daef4763015bf2" instance_seqno: 1786267925708887 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 09:32:06.151993 16047 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 09:32:06.154748 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.146739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59978)
0809 09:32:06.146853 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:06.146868 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:06.146944 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:06.147263 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:06.147266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:06.147466 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:06.147556 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:06.147563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.148495 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.148498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:06.149261 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:06.149270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.149851 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.149855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:06.149935 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.150412 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:06.150425 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:06.151643 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:06.153346 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:06.153350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:06.153352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:06.153536 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:06.153804 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:06.153807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:06.153808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:06.153907 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:06.154275 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:06.154278 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:06.154372 (+ 94us) client_negotiation.cc:769] Sending connection context
0809 09:32:06.154476 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 09:32:06.154600 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260809 09:32:06.154765 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.146810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59978 (local address 127.9.102.60:44991)
0809 09:32:06.147078 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.147082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.147094 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:32:06.147125 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.147127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.147171 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.147244 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.147683 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.148353 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.149403 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.149744 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.149831 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.151802 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:06.151805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:06.151807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:06.151832 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:06.153236 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.153630 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.153634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.153635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.153712 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.153995 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.153999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.154001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.154175 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:06.154281 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:06.154540 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 09:32:06.154645 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260809 09:32:06.157280 16040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d: Bootstrap starting.
I20260809 09:32:06.159214 16040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d: Neither blocks nor log segments found. Creating new log.
I20260809 09:32:06.159909 16040 log.cc:824] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d: Log is configured to *not* fsync() on all Append() calls
I20260809 09:32:06.161468 16040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d: No bootstrap required, opened a new log
I20260809 09:32:06.163990 16040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } }
I20260809 09:32:06.164216 16040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f8e1f8ed494e02a022ac5b9651d34d, State: Initialized, Role: FOLLOWER
I20260809 09:32:06.164496 16040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } }
I20260809 09:32:06.164902 16210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } } }
I20260809 09:32:06.165019 16210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:06.165519 16040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:32:06.166049 15906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } attempt: 3
Aug 09 09:32:06 dist-test-slave-7vjv krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786267925, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.102.62@KRBTEST.COM for kudu/127.9.102.60@KRBTEST.COM
I20260809 09:32:06.168447 15913 master_runner.cc:186] Error getting master registration for 127.9.102.62:43437: OK, instance_id { permanent_uuid: "825c670335614840b8daef4763015bf2" instance_seqno: 1786267925708887 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 09:32:06.171664 15913 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 09:32:06.172518 16231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:32:06.172631 16231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:32:06.174803 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.166362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.102.60:44991 (local address 127.0.0.1:59990)
0809 09:32:06.166502 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 09:32:06.166514 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0809 09:32:06.166579 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 09:32:06.166787 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 09:32:06.166790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 09:32:06.166974 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0809 09:32:06.167083 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:06.167089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.167929 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.167933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:06.168727 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 09:32:06.168737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.169338 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.169341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 09:32:06.169414 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.169863 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 09:32:06.169896 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 09:32:06.171419 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 09:32:06.173267 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:06.173270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:06.173272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:06.173503 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:06.173827 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 09:32:06.173829 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 09:32:06.173832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 09:32:06.173928 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 09:32:06.174366 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 09:32:06.174368 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 09:32:06.174463 (+ 95us) client_negotiation.cc:769] Sending connection context
0809 09:32:06.174551 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 09:32:06.174667 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260809 09:32:06.175336 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.166497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.9.102.60:44991)
0809 09:32:06.166582 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.166586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.166598 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:32:06.166641 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.166644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.166679 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.166767 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.167194 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.167824 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.168855 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.169170 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.169245 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.171575 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:06.171579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:06.171581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:06.171602 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:06.173139 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.173615 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.173617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.173619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.173720 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.174040 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.174044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.174045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.174234 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:06.174311 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:06.175086 (+ 775us) server_negotiation.cc:299] Negotiation successful
0809 09:32:06.175193 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260809 09:32:06.177987 15906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2: Bootstrap starting.
I20260809 09:32:06.180277 15906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2: Neither blocks nor log segments found. Creating new log.
I20260809 09:32:06.181017 15906 log.cc:824] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2: Log is configured to *not* fsync() on all Append() calls
I20260809 09:32:06.183408 15906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2: No bootstrap required, opened a new log
I20260809 09:32:06.186874 15906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } }
I20260809 09:32:06.187136 15906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825c670335614840b8daef4763015bf2, State: Initialized, Role: FOLLOWER
I20260809 09:32:06.187541 15906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } }
I20260809 09:32:06.188081 16234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } } }
I20260809 09:32:06.188273 16234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:06.189185 15861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 09:32:06.189348 15861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:32:06.190364 15861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } }
I20260809 09:32:06.191221 15861 leader_election.cc:288] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [CANDIDATE]: Term 1 election: Requested vote from peers 07f8e1f8ed494e02a022ac5b9651d34d (127.9.102.61:36479), e9f0aa934cd449e4940f962aa8e1b8f3 (127.9.102.60:44991)
I20260809 09:32:06.192065 15993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "825c670335614840b8daef4763015bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "07f8e1f8ed494e02a022ac5b9651d34d"
I20260809 09:32:06.192281 15993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:32:06.193459 15993 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825c670335614840b8daef4763015bf2 in term 1.
I20260809 09:32:06.193848 16123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "825c670335614840b8daef4763015bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3"
I20260809 09:32:06.194067 16123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 09:32:06.195227 16123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825c670335614840b8daef4763015bf2 in term 1.
I20260809 09:32:06.195580 15820 leader_election.cc:302] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 825c670335614840b8daef4763015bf2, e9f0aa934cd449e4940f962aa8e1b8f3; no voters:
I20260809 09:32:06.195844 16234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 09:32:06.196113 16234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [term 1 LEADER]: Becoming Leader. State: Replica: 825c670335614840b8daef4763015bf2, State: Running, Role: LEADER
I20260809 09:32:06.196139 15906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 09:32:06.196487 16234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } }
I20260809 09:32:06.198088 16234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 825c670335614840b8daef4763015bf2. Latest consensus state: current_term: 1 leader_uuid: "825c670335614840b8daef4763015bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } } }
I20260809 09:32:06.198447 16234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [sys.catalog]: This master's current role is: LEADER
W20260809 09:32:06.204788 16262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 09:32:06.204860 16262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 09:32:06.205124 16263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260809 09:32:06.205616 15826 tablet.cc:2369] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 09:32:06.205803 16263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 09:32:06.210103 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.200715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.9.102.62:43437)
0809 09:32:06.200817 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.200821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.201270 (+ 449us) server_negotiation.cc:407] Connection header received
0809 09:32:06.201473 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.201477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.201518 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.201566 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.202504 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.203245 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.204362 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.204664 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.204724 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.206915 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:06.206918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:06.206921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:06.206942 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:06.208445 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.208891 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.208894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.208896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.208967 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.209244 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.209247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.209249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.209423 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:06.209511 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:06.209828 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 09:32:06.209946 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 09:32:06.218650 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.200656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.9.102.61:36479)
0809 09:32:06.200770 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.200775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.210820 (+ 10045us) server_negotiation.cc:407] Connection header received
0809 09:32:06.211000 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.211003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.211062 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.211120 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.211883 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.212579 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.213579 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.213861 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.213920 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.215322 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:06.215325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:06.215328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:06.215350 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:06.216853 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.217328 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.217334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.217336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.217410 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.217750 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.217754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.217755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.218006 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:06.218107 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:06.218416 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 09:32:06.218514 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260809 09:32:06.222889 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:06.201347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.9.102.60:44991)
0809 09:32:06.201472 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.201476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.212314 (+ 10838us) server_negotiation.cc:407] Connection header received
0809 09:32:06.212479 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.212482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.212518 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.212572 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.213504 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.214136 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.215132 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:06.215415 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:06.215483 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:06.219612 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 09:32:06.219616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 09:32:06.219620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 09:32:06.219645 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 09:32:06.221254 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.221740 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.221743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.221745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.221822 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 09:32:06.222086 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 09:32:06.222089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 09:32:06.222090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 09:32:06.222261 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 09:32:06.222342 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:06.222663 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 09:32:06.222760 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 09:32:06.224951 16123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [term 1 FOLLOWER]: Refusing update from remote peer 825c670335614840b8daef4763015bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:32:06.225535 16242 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:32:06.228922 16184 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 825c670335614840b8daef4763015bf2. Latest consensus state: current_term: 1 leader_uuid: "825c670335614840b8daef4763015bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } } }
I20260809 09:32:06.229025 16184 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:06.230792 15993 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [term 1 FOLLOWER]: Refusing update from remote peer 825c670335614840b8daef4763015bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 09:32:06.231245 16242 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 09:32:06.234154 16210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 825c670335614840b8daef4763015bf2. Latest consensus state: current_term: 1 leader_uuid: "825c670335614840b8daef4763015bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } } }
I20260809 09:32:06.234259 16210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:06.241394 16234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "825c670335614840b8daef4763015bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } } }
I20260809 09:32:06.241503 16234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [sys.catalog]: This master's current role is: LEADER
I20260809 09:32:06.241851 16210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "825c670335614840b8daef4763015bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } } }
I20260809 09:32:06.241951 16210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:06.242277 16271 mvcc.cc:205] Tried to move back new op lower bound from 7316553425790967808 to 7316553425704128512. Current Snapshot: MvccSnapshot[applied={T|T < 7316553425790967808}]
I20260809 09:32:06.243193 16242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "825c670335614840b8daef4763015bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } } }
I20260809 09:32:06.243288 16242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2 [sys.catalog]: This master's current role is: LEADER
I20260809 09:32:06.245509 16184 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "825c670335614840b8daef4763015bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825c670335614840b8daef4763015bf2" member_type: VOTER last_known_addr { host: "127.9.102.62" port: 43437 } } peers { permanent_uuid: "07f8e1f8ed494e02a022ac5b9651d34d" member_type: VOTER last_known_addr { host: "127.9.102.61" port: 36479 } } peers { permanent_uuid: "e9f0aa934cd449e4940f962aa8e1b8f3" member_type: VOTER last_known_addr { host: "127.9.102.60" port: 44991 } } }
I20260809 09:32:06.245633 16184 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 09:32:06.246680 16276 mvcc.cc:205] Tried to move back new op lower bound from 7316553425790967808 to 7316553425704128512. Current Snapshot: MvccSnapshot[applied={T|T < 7316553425790967808}]
I20260809 09:32:06.246841 16272 mvcc.cc:205] Tried to move back new op lower bound from 7316553425790967808 to 7316553425704128512. Current Snapshot: MvccSnapshot[applied={T|T < 7316553425790967808}]
I20260809 09:32:06.250599 16263 catalog_manager.cc:1347] Generated new cluster ID: 243aa0f89a2e4951ac9a082567d1a1a4
I20260809 09:32:06.250682 16263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 09:32:06.278223 16263 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 09:32:06.279294 16263 catalog_manager.cc:1504] Loading token signing keys...
I20260809 09:32:06.290493 16263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 825c670335614840b8daef4763015bf2: Generated new TSK 0
I20260809 09:32:06.291222 16263 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 09:32:06.323544 16046 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.320799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.9.102.61:36479)
0809 09:32:06.320925 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.320929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.322625 (+ 1696us) server_negotiation.cc:407] Connection header received
0809 09:32:06.322786 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.322790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.322831 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.322889 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.323434 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41368: BlockingRecv error: recv got EOF from 127.0.0.1:41368 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260809 09:32:06.325047 15908 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.321490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.9.102.62:43437)
0809 09:32:06.321588 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.321591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.323993 (+ 2402us) server_negotiation.cc:407] Connection header received
0809 09:32:06.324371 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.324375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.324421 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.324468 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.324936 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39666: BlockingRecv error: recv got EOF from 127.0.0.1:39666 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260809 09:32:06.326062 16182 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.322029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.9.102.60:44991)
0809 09:32:06.322139 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.322144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.324297 (+ 2153us) server_negotiation.cc:407] Connection header received
0809 09:32:06.325377 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.325380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.325424 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.325468 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.325964 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60000: BlockingRecv error: recv got EOF from 127.0.0.1:60000 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260809 09:32:06.330072 16046 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.328744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.9.102.61:36479)
0809 09:32:06.328823 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.328828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.329247 (+ 419us) server_negotiation.cc:407] Connection header received
0809 09:32:06.329430 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.329433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.329496 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.329542 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.329998 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41370: BlockingRecv error: recv got EOF from 127.0.0.1:41370 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260809 09:32:06.334640 16046 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.333251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.9.102.61:36479)
0809 09:32:06.333368 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.333371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.333857 (+ 486us) server_negotiation.cc:407] Connection header received
0809 09:32:06.334028 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.334031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.334072 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.334116 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.334570 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41376: BlockingRecv error: recv got EOF from 127.0.0.1:41376 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260809 09:32:06.349319 16046 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.347794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.9.102.61:36479)
0809 09:32:06.347897 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.347900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.348478 (+ 578us) server_negotiation.cc:407] Connection header received
0809 09:32:06.348644 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.348647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.348689 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.348734 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.349223 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41382: BlockingRecv error: recv got EOF from 127.0.0.1:41382 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260809 09:32:06.356190 16046 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.355181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.9.102.61:36479)
0809 09:32:06.355289 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.355292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.355475 (+ 183us) server_negotiation.cc:407] Connection header received
0809 09:32:06.355561 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.355564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.355607 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.355692 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.356116 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41394: BlockingRecv error: recv got EOF from 127.0.0.1:41394 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260809 09:32:06.366848 16046 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.365864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.9.102.61:36479)
0809 09:32:06.365958 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.365962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.366163 (+ 201us) server_negotiation.cc:407] Connection header received
0809 09:32:06.366239 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.366256 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.366308 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.366376 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.366763 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41404: BlockingRecv error: recv got EOF from 127.0.0.1:41404 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260809 09:32:06.385210 16046 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.384426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.9.102.61:36479)
0809 09:32:06.384521 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.384525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.384635 (+ 110us) server_negotiation.cc:407] Connection header received
0809 09:32:06.384722 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.384724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.384757 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.384816 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.385123 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41410: BlockingRecv error: recv got EOF from 127.0.0.1:41410 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260809 09:32:06.419781 16046 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.418932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41426 (local address 127.9.102.61:36479)
0809 09:32:06.419060 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.419063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.419134 (+ 71us) server_negotiation.cc:407] Connection header received
0809 09:32:06.419226 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.419228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.419264 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.419326 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.419692 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41426: BlockingRecv error: recv got EOF from 127.0.0.1:41426 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 09:32:06.486641 16046 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.485657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.9.102.61:36479)
0809 09:32:06.485798 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.485802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.485902 (+ 100us) server_negotiation.cc:407] Connection header received
0809 09:32:06.486006 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.486008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.486057 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.486124 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.486523 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41436: BlockingRecv error: recv got EOF from 127.0.0.1:41436 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260809 09:32:06.617864 16046 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.616786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.9.102.61:36479)
0809 09:32:06.616925 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.616929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.617121 (+ 192us) server_negotiation.cc:407] Connection header received
0809 09:32:06.617242 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.617244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.617290 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.617367 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.617770 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41440: BlockingRecv error: recv got EOF from 127.0.0.1:41440 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 09:32:06.876719 16046 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 09:32:06.875752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.9.102.61:36479)
0809 09:32:06.875864 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:06.875867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:06.875902 (+ 35us) server_negotiation.cc:407] Connection header received
0809 09:32:06.876032 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:06.876034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:06.876074 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:06.876213 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 09:32:06.876578 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41448: BlockingRecv error: recv got EOF from 127.0.0.1:41448 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 09:32:07.150210 16207 catalog_manager.cc:1259] Loaded cluster ID: 243aa0f89a2e4951ac9a082567d1a1a4
I20260809 09:32:07.150291 16207 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3: loading cluster ID for follower catalog manager: success
I20260809 09:32:07.151749 16207 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3: acquiring CA information for follower catalog manager: success
I20260809 09:32:07.152926 16207 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e9f0aa934cd449e4940f962aa8e1b8f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 09:32:07.173624 16231 catalog_manager.cc:1259] Loaded cluster ID: 243aa0f89a2e4951ac9a082567d1a1a4
I20260809 09:32:07.173691 16231 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d: loading cluster ID for follower catalog manager: success
I20260809 09:32:07.174948 16231 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d: acquiring CA information for follower catalog manager: success
I20260809 09:32:07.175819 16231 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 07f8e1f8ed494e02a022ac5b9651d34d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 09:32:07.394208 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:07.390630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.9.102.61:36479)
0809 09:32:07.390847 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:07.390851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:07.390963 (+ 112us) server_negotiation.cc:407] Connection header received
0809 09:32:07.391152 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:07.391155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:07.391277 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:07.391366 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:32:07.391582 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:07.392045 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:07.392826 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:07.393031 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:07.393113 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:07.393364 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:32:07.393808 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:32:07.393897 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:07.393953 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 09:32:07.393997 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
I20260809 09:32:07.400695 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:07.397111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.9.102.60:44991)
0809 09:32:07.397370 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:07.397376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:07.397393 (+ 17us) server_negotiation.cc:407] Connection header received
0809 09:32:07.397582 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:07.397584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:07.397684 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:07.397794 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:32:07.397994 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:07.398498 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:07.399364 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:07.399571 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:07.399653 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:07.399902 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:32:07.400282 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:32:07.400400 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:07.400472 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 09:32:07.400549 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":110,"threads_started":1}
I20260809 09:32:07.407006 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:07.403815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60016 (local address 127.9.102.60:44991)
0809 09:32:07.403935 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:07.403940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:07.403953 (+ 13us) server_negotiation.cc:407] Connection header received
0809 09:32:07.403990 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:07.403992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:07.404041 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:07.404125 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:32:07.404418 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:07.404857 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:07.405982 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:07.406260 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:07.406352 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:07.406625 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:32:07.406683 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:32:07.406747 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:07.406860 (+ 113us) server_negotiation.cc:299] Negotiation successful
0809 09:32:07.406902 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 09:32:07.407136 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:07.403184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.9.102.61:36479)
0809 09:32:07.403295 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:07.403299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:07.403714 (+ 415us) server_negotiation.cc:407] Connection header received
0809 09:32:07.403872 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:07.403876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:07.403921 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:07.403967 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:32:07.404317 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:07.404972 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:07.406037 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:07.406284 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:07.406380 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:07.406720 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:32:07.406787 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:32:07.406828 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:07.406977 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 09:32:07.407018 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260809 09:32:07.407598 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 09:32:07.403139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.9.102.62:43437)
0809 09:32:07.403453 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0809 09:32:07.403456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 09:32:07.403468 (+ 12us) server_negotiation.cc:407] Connection header received
0809 09:32:07.403650 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 09:32:07.403656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 09:32:07.403820 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 09:32:07.403927 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 09:32:07.404164 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:07.404890 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:07.405822 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 09:32:07.406101 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 09:32:07.406190 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 09:32:07.406460 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 09:32:07.406732 (+ 272us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 09:32:07.406811 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 09:32:07.406873 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 09:32:07.406939 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":61,"threads_started":1}
I20260809 09:32:07.410614 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 15794
I20260809 09:32:07.424587 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 15913
I20260809 09:32:07.434444 9624 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taska20N0I/build/debug/bin/kudu with pid 16047
2026-08-09T09:32:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3136 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6400 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (280751 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST